A: Assuming you intend b = m^2 - n^2, you can substitiute the given expressions for "a" and "b" and rearrange the equation to get the desired result.
... b = a + 1
... m^2 -n^2 = 2mn + 1 . . . . . substitute for a and b
... m^2 -2mn - n^2 + 2n^2 - 2n^2 = 1 . . . . . subtract 2mn, add 2n^2 - 2n^2 on left
... (m -n)^2 - 2n^2 = 1 . . . . . . .simplify to get the desired form
B: This is a Pell equation of the form
... x^2 -2y^2 = 1 . . . . . . . . . . . . where x = (m-n) and y = n
It has solutions that can be derived from the convergents of the continued fraction expansion of √2. That continued fraction is of the form
... √2 ≈ 1 + 1/(2 + 1/(2 + 1/(2 + ...))) . . . . where the denominator fractions continue in the same pattern. The first few convergents are
... 1, 3/2, 7/5, 17/12, 41/29, 99/70, 239/169, ...
Every other one gives rise to a solution. For example, 3/2 tells you x=3, y=2, so
... m=3+2=5, and n=2.
The second-smallest solution is x=17, y=12, so
... m=17+12=29, and n=12.
The corresonding two smallest triangles are
... {a, b, c} = {20, 21, 29}, {696, 697, 985}