I believe the answer is: The warriors weren't scared because they didn't think that a woman could fight with as much strength, power and courage that men fight with.
The warriors did not know that Grendel's mother is giant man-eating humanoid beast who could devoured several men in one single day , posses superhuman strength and her body is basically impenetrable by weapons. when the warriors realized it they're too late.