The United States and the Soviet Union were in the middle of a Cold War. To have their enemies rockets so close to home is devastating.
The missiles on Cuba meant that the US was of the striking range and the threat of nuclear war increased exponentially. The Soviets could use those rockets to attack the US and they could not have time to react. Defending against that kind of force, so close to the border would be a nightmare and would certainly lead to great loss of life.