with calm exasperation
The narrator is not happy that he has been tricked into Simon Wheeler's story-telling antics. He is annoyed that he is stuck listening to these tales about Smiley's gambling. While he listens well, he is not amused, patient, or attentive.