The narrator/speaker asks his lady love to presume that she has grown old and grey and is sitting by the fire nodding. Then he asks her to read from her book of memories and reminisce her past when she was in her prime youth. While she is thus engaged in recalling her past, the poet reminds her that though she undoubtedly had a great many suitors who admired her beauty and elegance and professed ‘love’ which may be true or false, he alone loved her unconditionally.
He says that he loved her inner beauty and even the fading away of her youth and beauty. The phrase ‘how many’ in the first line (second stanza) stands in contrast to ‘But one man’, in the third line (second stanza). While many suitors loved her beauty and elegance, he alone loved her pilgrim soul as well as the sorrows of her changing face.