The explanation of the problem is shown in the attached figure.
Let the side length of the square be x.
The triangular regions outside the square are all isosceles right triangles
The length of the hypotenuse is 3 in
So,
The length of each leg = 3/√2
The area of the given triangle = (1/2) * (3/√2)*(3/√2) = 9/4
With the help of figure:
Area1= (1/2) * (x/√2)*(x/√2) = x²/4
Area2= (1/2) * (x)*(x) = x²/2
Area3= (1/2) * (x)*(x) = x²/2
Area of the square = x²
So, Area1 + Area2 + Area3 + square area = 9/4
∴
x²/4 + x²/2 + x²/2 +x² = 9/4
(9/4) x² = 9/4
x² = 1
∴ x = 1
∴ The side of the square which is inscribed in a right isosceles triangle = 1 in