I read the poem and found that the line with most underlying tones of regret is "Back then it never fell so solemnly"
But now I am mostly at the window
watching the late afternoon light.
Back then it never fell so solemnly
against the side of my tree house,
Billy Collins is saying that he previously watched the afternoon light, but in the past it was not as solemnly as it is now.