Answer:
3 in
Explanation:
A graphing calculator is useful for finding the roots of a cubic equation. The one rational root is x=3. The other two solutions are irrational. As it happens, one of them is about 2.89, also rounding to 3 when rounded to the nearest inch.
The side length of the square cutout is 3 inches.
_____
The equation can be put into standard form as ...
x^3 -18x^2 +80x -105 = 0
Possible rational roots are 1, 3, 5, 7 (divisors of 105). We can quickly rule out 1, since the sum of coefficients is not zero. Trial of 3 by synthetic division shows it to be a rational root, so the factorization is ...
(x -3)(x^2 -15x +35) = 0
The solutions to the quadratic can be found by any of the usual methods. We can rewrite it to ...
x^2 -15x +35 = (x^2 -15x +56.25) -21.25 = 0 = (x -7.5)^2 -21.25
x = 7.5 ±√21.25 = {2.89, 12.11}
As we said above, the x value of 2.89 rounds to 3 anyway.
The side length of the cut square is 3 inches.
_____
Check
Volume = x(16 -2x)(20 -2x) = 3(10)(14) = 420 . . . cubic inches.