The most likely answer for this question would be internal revolutions and revolts.
This would probably be applicable not just to an empire but to most growing communities. Having a high agricultural potential would mean that the sustainability of their growth is high. An adequate military resources would assure the safe growth of the community or empire. State and local governments would maintain and oversee the growth into its utmost potential and the efficiency of the growth be considered in future plans Internal revolutions and revolts, however, would result to damages not just to the physical structure of the empire, but also to its people. Thus, not just having the whole thing back to square one, but also slowing the progress that is eventually made.