The maximum volume is shown by a graphing calculator to be about 1056.3 in^3.
_____
That volume is achieved by folding the edges up 3.924 inches.
The volume as a function of box depth is
.. V = x(30 -2x)(20 -2x)
.. = 4x^3 -100x^2 +600x
Differentiating, we have
.. V' = 12x^2 -200x +600
We want to find where this is zero.
.. x = (200 ±√((-200)^2 -4(12)(600)))/24
.. = (200 -40√7)/24 . . . . . only the solution less than 10 makes sense
.. = (1/3)(25 -5√7)
Putting this into the formula for the volume, we find the volume of the box to be
.. V = (1000/27)*(10 +7√7) ≈ 1056.3 in^3