This can be factored out with the form: (ax+by)^2 = a^2x^2 + 2abxy + b^2y^2
So by comparison, you will have 3 equations:
a^2 = 10 2ab = 17 b^2 = 3
You will notice that these 3 equations cannot be satisfied at the same time, which means it cannot be factored out perfectly
So, you can only satisfy two of them, and compromise the 3rd one.
Let's say I want to use the first and the last one, i.e. a = sqrt(10) and b = sqrt(3)
Then, (sqrt(10)x + sqrt(3)y)^2 = 10x^2 + 2sqrt(30)xy + 3y^2
But you need 17xy in the question.
So your final answer will be (sqrt(10)x + sqrt(3)y)^2 + (17xy - 2sqrt(30)xy)
You can use another 2 pairs and get different forms too.