If the pressure inside the balloon is too high relative to the pressure outside, the balloon will burst. If you don't overinflate it, the pressures inside and outside the balloon should be equal, so there is no net pressure on the balloon. At high altitudes, though, the atmospheric pressure decreases, but the pressure inside the balloon remains the same - now there is a net pressure outwards on the balloon, which, if sufficiently high, will burst it.