Answer: Choice D) 18/35
===========================================
Step-by-step explanation:
There are 3 ways to select the single math book and 4*3/2 = 12/2 = 6 ways to pick the two other books that are either physics or history (order doesn't matter). This is effectively because we have 3+1 = 4 books that are either physics or history, and we're using the nCr combination formula.
Overall, there are 3*6 = 18 ways to select the three books such that one is math, and the other two are either physics or history.
-------------------
There are 3+3+1 = 7 books total. Since we're selecting 3 of them, we use the nCr formula again and you should get 35.
Or you could note how (7*6*5)/(3*2*1) = 210/6 = 35
This says there are 35 ways to select any three books where we can tell the difference between any subject (ie we can tell the difference between the math books for instance).
-------------------
We found there are 18 ways to get what we want out of 35 ways to do the three selections. Therefore, the answer as a fraction is 18/35