To figure out which of the coordinates don't work, we need to substitute their values of x and y to figure out if they work or not. We will know they don't work if the result we get is something crazy like 2=25.
Substitute the values of x and y from a

Now we simplify
Since this is true we know that the point (5,6) works. Now let's move onto B.
Once again, we substitute.

This simplifies to

Since this is obviously not right, we know that is the point that doesn't work in this system of equations.