It is mainly because we have chemical compounds named Hydrides - compounds which consists of metal ions and hydrogen ion
. So, hydrogen ion is used to "take" electron more than "give" it, as we also can see in organic compounds. Third reason is that its standard electrode potential is further higher than metals' potentials, which obviously makes it nonmetal.
Hope this helps!