The thing is, people were not really better off as hunter gatherers. Hunter gatherers were never sure if there would be enough food: in the winder, or when they had bad luck, they would not eat and the weak could starve. That's why they were so happy to change to agriculture.