People from the Union and Confederacy said this because a lot of the people fighting in the war were farmers and the lower class since they couldn’t afford to exempt. They either didn’t have enough slaves or neither did they have the money to do so. In the reading it says “any man who owned more than 20 slaves could not be drafted”. Only the rich plantation owners would have the kind of money to buy that many slaves. And in the Union you had to “pay 300 dollars not to serve at all”. During that time, 300 dollars was a lot compared to how much it’s worth today, hence people calling it “a rich man’s war and a poor man’s fight”.