Answer:
Well, I guess in a formal sense you could call relational algebra a type of logic, which puts it into the realm of ‘math’, though I don’t think most mathematicians would look at it as being a very significant area, more ‘applied logic’ than anything else.
That being said, relational algebra DOES present some formalisms in the shape of relations, tuples, normalizations, and denormalization, etc. There is also a model of sequential operations and thus the underpinnings of the ACID database concept.
Since the overwhelming primary use of RA has been as a way to structure our ideas about SQL RDBMSes there isn’t really a lot of deep theory there. I don’t think anyone has really built any other models of RA beyond an RDBMS. There have been limited non-SQL RDBMS implementations, but AFAIK none of them are really actively maintained today.