At the beginning of the poem, he says why he does not love her. He doesn't love her as if she was a salt-rose, topaz, or carnation, but he loves her as if she was a plant that does not bloom. This means that he does not love her superficially, but he loves her deeply for who she is on the inside.
Flowers are beautiful things that everyone admires, but no one would admire a flower that doesn’t bloom because they could not see the beauty that it contains.