That statement is true.
Slavery at pre-civil war was only an acceptable practice in southern states. From those southern states, only the people who are rich enough to own their own plantation could buy a slave. So, the number of people who owned slave probably less than 10% of total population.