The explanation is actually pretty simple.
The human hearing machinery doesn't hear ALL vibrations. Some are too soft. And some vibrate too slow or too fast.
Our ear-brain hearing system only registers a sound from waves in the air that vibrate AT LEAST about 20 times per second but LESS than about 20,000 times per second.
Yes. When you move your hand back and forth through the air, you certainly do create a "longitudinal wave" of air pressure that spreads out in all directions, and some of it does enter your ear. BUT ... There's no way you can move your hand back and forth more than 20 times per second.
So although your ear-brain system may be aware of the pressure wave, it just ignores it, and doesn't interpret that as a sound.