One big reason was the fact that Rome had the bad luck of a number of extremely bad emperors who didn't really care that much about Rome becoming and staying a strong empire that would last the ages, what they did care about were hedonistic pleasures and similar issues. Not very relevant to leading an empire. Two such emperors are Caligula and Neron, both very known for their excesses and brutality as well!