It is not specified to whom the Pardoner is speaking when he says "My Lords..." It is possible that he is addressing a group of people, such as a group of aristocrats or members of the clergy, using a formal title of respect. Alternatively, the Pardoner may be using the phrase "My Lords" as a way of addressing anyone in a position of authority or power, regardless of their actual title. In this case, the Pardoner may be using the phrase as a way of flattering or ingratiating himself to those he is speaking to, in an effort to persuade them to purchase his indulgences or support his cause.