Final answer:
Three sets of values for a and b that satisfy the coplanarity of the given vectors are a = 4 and b = -13, a = 16 and b = 45, and a = -12 and b = -32.
Step-by-step explanation:
The vectors (5,2,1), (-1,3,6) and (a, b, 5) are coplanar if and only if the scalar triple product of these vectors is equal to zero.
Let's calculate the scalar triple product:
(5,2,1) · (-1,3,6) · (a, b, 5) = 0
This simplifies to -29a + 9b - 41 = 0
From this equation, we can determine three sets of values for a and b that satisfy it. One solution is a = 4 and b = -13. Two other valid solutions are a = 16 and b = 45 or a = -12 and b = -32.