The easiest way to do this is to find the number of square feet each way would include, and then pick the largest one.
10 x 40 = 400 square feet
15 x 30 = 450 square feet
18 x 24 = 432 square feet
30 x 30 = 900 square feet
HOWEVER, he only has 60 feet to make three sides with, since his house will be the fourth side. That means not all of the above options will work.
40 + 10 + 10 = 60
30 + 15 + 15 = 60
24 + 18 + 18 = 60
30 + 30 + 30 = 90
That means that option D is not the answer, since he doesn't have enough fence to make that big of an area. Now if we look back at the first part, we will see that the most square footage comes from the 15 by 30 area.
Answer: B. 15 ft by 30 ft.
Hope that helps!