Because inside they know he is right that everyone has that Secret sin that they aren't willing to share with the world because they are afraid of getting shamed. because the minister wore the black veil it represented the secret sins people were living behind. Everyone wants to be perceived as holy and that they don't sin, when in reality just thinking about doing bad things is a sin so they are unable to admit their secret sins