First, it may be important to identify what is meant by a negative mass. In particular, it is necessary to distinguish negative mass from anti-matter. Anti-matter is matter made up of atomic particles that have opposite electrical charges from those of ordinary matter. Anti-matter, as we know it, does not have negative mass. Anti-matter has been shown to exist in experiments, but there is no evidence that any anti-particles have negative mass. In fact, the idea that anti-particles, such as the positron, are regular particles, such as the electron, going backwards in time is probably no more than a convenient fantasy, since both positrons and electrons have positive mass. This can be demonstrated by the physics of the mutual annihilation of a positron and an electron, which creates energy in the form of photons, that is devolved by E = mc^2, to the sum of the masses of the two particles. In such collisions, charge is annihilated, but mass-energy is not.
If there were some form of matter with a negative mass, how would it behave? The first insight is that it would move in a direction opposite to that in which it is pushed. So, if we have a lump of mass of, say -3 grams, and we push it (i.e. apply a force) to the right, it will move (i.e. accelerate) to the left. The harder we try to push it to the right, the more fervently it would move to the left. This follows from Newton's Law, F = ma, which clearly shows that if a positive force, F, is applied to a negative mass, -m, the acceleration, a, must be negative (i.e. opposite to the direction of the force) for the equation to hold.
Even if one asserts that the Newton equation: F = ma is the inertial definition of mass (i.e. m = F/a), then if a piece of matter accelerates in a direction opposite to the force applied to it, then its mass is negative.
But, how would a negative mass behave in the Earth's gravity? Some may be surprised to know that a lump of negative mass, would fall down at the same rate as any positive mass. The influence of gravity is an acceleration, not a force. This was known by Galileo, who demonstrated that objects of different masses accelerate toward the earth at the same rate regardless of their size or weight in his legendary experiment from the Tower of Pisa. Perhaps it is unfortunate that we have come to speak of the "force" of gravity, when we should speak of the "acceleration" of gravity
However, in order to restrain a negative mass from falling down, we would need to apply a downward force to it. Thus, it would rest under a table, rather than on top of one. Unfortunately, the balance of forces could not stay in equilibrium, for any small perturbation in the restraining force, such as from a puff of wind, would cause the object to either fall or to bore its way upward through the table-top. Once it hits the floor, the upward force exerted by the floor on this object would result in its downward acceleration that would cause it either to disintegrate or to bore its way all the way to the center of the Earth.
If a negative mass were resting in equilibrium under a table and the table were on a scale, the weight of the table would be reduced by the negative mass under it, just as it is increased by the weight of ordinary masses on top of it. Even if the negative mass under the table had a greater absolute value of mass than the table itself, the ensemble would be restrained by holding it down, but when let go it would still fall downward because the free-fall acceleration is in the direction of the restraining force, rather than opposite to it as in ordinary matter.
If an entire planet were made of negative mass, its gravitational acceleration would be upward from its surface rather than downward. All objects, of either positive or negative mass would be accelerated away from such a planet, and it would self-disintegrate. Even if atomic forces, such as those that hold atoms together in crystals were to oppose the disintegration of the rocks on the planet, each negative mass atom would be met by an ever stronger acceleration opposite to these attractive forces, with the consequence that all negative-mass matter would disintegrate into separate atoms, and perhaps into subatomic particles.
The orbital mechanics of negative mass are also unstable. If the Moon were made of negative mass, it could orbit the Earth, but the orbit would be around a center of mass that lies on a point that is along the extension of the line joining the centers of mass, and not between them, but at a point opposite the Earth from the Moon. The radius of the Moon's orbit would be greater than the distance between the Earth and Moon, rather than less. See the figure and math below. Any small perturbation of this orbit would cause the Moon to fly away or crash into the Earth.
If two planets of equal mass, one normally positive, and the other made of negative mass were to orbit each other, their radius of orbit around their center of gravity (cg) would be infinite. This follows from the singularity in the equation for the orbit radius for the Earth around the cg when the mass of the Moon and the mass of the Earth are equal to each other. In this limiting case, although in an unstable equilibrium, they would theoretically move at great speed through space in parallel straight lines!
Clearly, negative-mass matter is an untenable concept in Newtonian Physics. Negative-mass matter will not be found resting among the wads of chewing gum under tables. It will not be found orbiting the Earth or Sun, nor is it likely to be found at the center of the Earth or Sun. If it exists, it would self-disintegrate from an unstable equilibrium of internal forces. Matter, as-we-know-it, cannot have negative mass.
However, if we accept the equivalence of mass and energy through the equation , E = mc^2, there is an equivalent negative mass associated with the energy of a gravitational field.
In a gravitational field, the situation is the reverse. By examining the way in which one can add energy to a "gravitational capacitor" consisting of a spherical mass shell by expanding its radius, one can see that the gravitational field is reduced, rather than increased. If the potential energy of the shell is released by allowing the shell to collapse, the volume of gravitational field around the shell increases as the mass fills the void that had previously existed inside the shell.
One way to determine the energy density of a gravitational field is to imagine a perfect thin spherical shell of mass. This shell has a very small thickness relative to its outer radius, The gravitational field outside of this spherical shell is a field of acceleration toward the center of the shell, whose strength diminishes inversely as the square of the distance from the center of the shell. If the mass of the shell is rearranged within its radius while remaining spherically symmetrical, the gravitation field beyond that radius will not change. However the gravitational field within this radius will depend upon the energy released as the mass is allowed to assume a rearrangement.
Once the energy density of a gravitational field is known, then one can determine the negative mass contribution to the mass of an object, such as a spherical mass of uniform density as follows:
This same expression for the negative contribution of mass due to gravitational fields has been reported by Prof. Robert H. Dicke in Gravitation and the Universe , in his publication of Jayne Lectures for 1969, and published by the American Philosophical Society in 1970. It is shown as ratio of gravitational energy to total energy for an ordinary laboratory sized object in his equation (13) on page 23. Dicke, however does not show it with a negative sign there, but acknowledges on page 25 that a body does have a negative gravitational energy which serves to reduce the gravitational acceleration associated with its mass.
Use this link to see some of the speculative consequences of the negative mass associated with gravitational fields. See what the magnitude of the gravitational negative mass is for planets, stars, clusters, and galaxies, and speculations on the negative gravitational mass of the entire universe, and black holes.
Links to other pages dealing with negative mass are listed below: