Answer: Choice C)
5*sqrt(3) - 20*sqrt(2)
Check out the attached image to see how I got that answer. Specifically look at figure 1. The other figures are supplemental steps to help break things up into a more organized solution (and to prevent too much clutter).
The idea I'm using to simplify is to basically factor the radicand (stuff under the radical or square root) into two numbers where one number is a perfect square. The goal is to pick the largest perfect square factor possible. Then I used the rule that sqrt(x*y) = sqrt(x)*sqrt(y) to break up the roots.
After each piece is simplified, the next step is to combine like terms. In this case, the sqrt(2) terms are combined.