Answer:
I think Chillingworth may have seen the letter "A" on Dimmesdale's chest.
Step-by-step explanation:
Well since Hester has one on her chest I think that it makes sense. If it's true, it means that Dimmesdale has had some kind of "interactions" with Hester which would lead me to think that he might as well be the secret man Hester doesn't want to reveal.
Maybe my explanation was too exaggerated...but I tried.