No. Jinnie is not afraid of the fisherman. Bit if he is threatened, he may be afraid.
Usually, at first, the fisherman pleads for his life, but the jinni would not concede. The fisherman quickly put the cap back on and threatened to throw it back to the sea. The jinni pleaded with the fisherman.
Example- "The Wazir and the Sage Duban" as an example of why the jinni should have spared him.