Oh but we do !
When you're some distance from the Earth and you're released, you and the ground accelerate toward each other, because of the mutual gravitational forces of attraction. The Earth's mass is a gazillion times YOUR mass, of course, so the Earth's acceleration is only one gazillionth of YOUR acceleration, but the Earth does accelerate toward you in that situation.
The ground does come toward you. The whole process is referred to as "falling".