I believe the answer is: Southerners feared it would lead to the end of slavery.
By giving the congress with the power to control the commerce, the congress could create a regulation regarding the type of labours that the southerners can use. When this happen, the Southerners believe that the Congress would leverage the power to eliminate slavery which they relied on for their plantation and could destroy their economy.