We can infer that the Pardoner is a wealthy man. However, we can also infer that he became rich by swindling people of their money - he sells them pardons for their sins, so each time they commit a sin, they can just come to him, buy a pardon, and all of a sudden, they are absolved of all their wrongdoings.