Answer: 18/35
===========================================
Step-by-step explanation:
There are 3 ways to select the single math book and
ways to pick the two other books that are either physics or history (order doesn't matter). This is effectively because we have
books that are either physics or history, and we're using the nCr combination formula.
Overall, there are
ways to select the three books such that one is math, and the other two are either physics or history.
-------------------
There are
books total. Since we're selecting 3 of them, we use the nCr formula again and you should get 35.
Or you could note how

This says there are 35 ways to select any three books where we can tell the difference between any subject (ie we can tell the difference between the math books for instance).
-------------------
We found there are 18 ways to get what we want out of 35 ways to do the three selections. Therefore, the answer as a fraction is 18/35