Answer:
He felt afraid even when the brothers weren't around.
Step-by-step explanation:
"Even if they weren't chasing me in FLESH (physically chasing him), they sure were chasing me in SPIRIT (mentally chasing him, meaning he could feel them but not see them, nor were they physically around them. He may have been a bit paranoid)."