Hmm you see this I a good question.
In my mindset they were looked down upon maybe even worse than the slaves themselves. You see how many slaves you owned was a indicator of your wealth the more you had the richer you were the less you had the poorer you were.
They were probably shunned in their communities and thought of as slave lovers. Treated badly and strongly Influenced by everyone's opinions that if you didn't have slaves you weren't anyone.
Hope I gave you some insight -beanz :)