"I don't know you from a hole in the wall" is like when you don't know who someone is just by looking at them
Step-by-step explanation:
It basically means he remembered the person from their jacket, not from how they look. I think he said this because he wanted the person to know that even though they might've been friends for a while or worked together, he wouldn't even recognise them if he saw them wearing a different outfit. He was able to identify the person by their jacket.
That's my opinion:)!