The correct option is B: new, overwhelming.
Adjectives are usually used to modify nouns. In this case, in the sentence "[although] the new students worked hard, they could barely keep up with the overwhelming amount of homework", "new" modifies "students" and "overwhelming" modifies "amount of homework".
The other options contain "barely" which is an adverb, not an adjective. Moreover, "hard" is also used as an adverb in the sentence. Adverbs such as "barely" modify a verb. That is to say, they add information about HOW an action is.