The people who were firmly against James K. Polk's expansionist program, often referred to as Manifest Destiny, were those who were strongly opposed to slavery. In Polk's plans of expanding and taking over more things, he included slave labor as a major factor in developing areas and financial sectors like cotton farms. Those who were against slave labor were opposed to the ideas of expanding the US on the backs of slaves.
