The correct answer is A. By late afternoon, we were worried to hear the reports of the storm.
Infinitive is termed as the third type of verbal. It can work as three parts of speech. Worried is the adverb here. It modifies the meaning of worry how people got worried when they got the storm report.
A good example of an infinitive phrase which acts as an adverb is, to make the matters worse, a strong wind propelled the fire.