Basically, the problem was deeply ingrained racism of white people, and the fact that they did not want to give up on their privileges (which led them to have house maids and servants, so they didn't have to work as hard as the black population, they could have physically easier jobs).