In Chapters 7-17 of "To Kill a Mockingbird," Jem becomes more curious about the Radley family, and he starts to develop empathy towards them. Jem initially took the blanket from the Radleys' porch to keep Scout warm during the fire at Miss Maudie's house. However, Jem now feels guilty for taking the blanket without permission and fears that it would be wrong to return it directly to the Radleys.
Jem says, "Don't put it on the Radleys, Atticus. Please, sir. Don't do that. Tell him we've got it and we'll return it to him. Promise that, Atticus. Promise you'll do that" (Chapter 7). Jem's reluctance to return the blanket to the Radleys directly suggests that he is aware of the social stigma attached to the family and fears that returning the blanket could cause more trouble for them. Additionally, Jem might be afraid of confronting the Radleys because he is aware of the rumors surrounding Boo's behavior.