If they remained in the south their emancipation actually left them worse off. Before the war they were treated as a valuable asset whose health and welfare the owners had to protect.
After the war, the owners were no longer obligated to treat them with care or respect and forced them to live on an inhumane amount of money. Those that objected could easily be fired and replaced with a more docile employee,
The North did better but living conditions were still pretty awful. Freedom brought its own set or problems. Industry was more than willing to take advantage of the poor. If that was not the case, why did it take until 1965 for the civil rights movement to gain real meaningful conditions for the colored?
There were economic improvements made between 1872 and 1965, but it should not have taken that long.