OK, here's what I've got:
Let x=one side of the square to be cut out of the corner
length of box=12-2x
width of box=8-2x
height of box=x
Volume=lwh
V(x)=(8-2x)(12-2x)(x)
=(96-16x-24x+4x^2)x
=(96-40x+4x^2)(x)
=96x-40x^2+4x^3
V'(x)=96-80x+12x^2
Set V'(x) to 0 to get the max volume, and use the quadratic formula to solve, but I'm not getting the right value for "x" according to the answer key. Have I set the problem up incorrectly? Thanks so much.