The speaker addresses Jeanne.
The first line of the poem is "Let us not talk philosophy, drop it, Jeanne". In this line, the speaker is talking to Jeanne and asking her not to talk about philosophy anymore. The speaker addresses her again in the fifth stanza when he says, "You are right, Jeanne,".