In this excerpt, Yeats is comparing the love a woman has received from many men, to that which she received from only one man (allegedly, the speaker). He argues that other men loved the woman when she was beautiful and graceful ("loved your moments of glad grace" and "loved your beauty"). However, only one man has loved her to the depths of her soul ("loved the pilgrim soul in you") and even when her beauty was fading and her face changing ("loved the sorrows of your changing face"). The love of this one man, therefore, is much deeper and much more real than that of any other one.