The answer is nutrition. Much like the difference between the height of people today and our grandparent's generation. Improved nutrition among wealthy families means that the bodies of the members of those families could reach their full potential height as limited by their genetics, and environmental factors did not come into play. In other words, the heights of poorer people are limited by both nurture and nature, whereas those of wealthyl people are limited only by nature.