Adding more territories to the south of the United States would create potential for the expansion of slavery, and an increase in the number of slave state representatives and senators in Congress, which would upset the contemporary balance of free to slave states.
