The way you can tell to use “is” and not “are” is to consider if “mathematics” is singular or not. “Mathematics” is typically used as a singular noun, to describe the field of study (not a group of mathematicians). However, it is plural in form- we describe said field of study as “mathematics” and not “mathematic”. This contrasts other singular nouns; biology, for instance, is not called biologies. Even though “mathematics” is plural in form, it is still singular. Thus, the most grammatically correct sentence would be “mathematics is very hard”. In my experience, however, it’s more common to say “math / maths is very hard.”