One of the main reasons why he was so controversial is because he was a slave owner, and made a lot of profit from holding slaves. He even brought slaves in the white house to work for him. Theres also track record of him physically abusing his slaves.
He played a big role in the westward movement of the US, but the outcome of that was removing natives (Indian Removal Act) from their land and causing a lot of deaths.