At that time, the most controversial decision made by Andrew Jackson would most likely be his efforts to move the Native Americans from their land.
He was a prominent supporter of making legislation that enable American military to take the native american land by force.
If, other representatives at that time let Jackson getting his way in politics, Future presidents would most likely had to deal with the after effects of ethnic cleansing initiated by Jackson, and the racial tension in united states might be even worse that it is today.