The surface area of the box is being referred to as A
The surface area will be the sum of the four rectangular sides' areas and the base area.
Base area = x²
Rectangular side area = xh
A = x² + 4xh
Volume is, as you said,
V = x²h
Lets substitute h from the formula of A to obtain A in our final answer. Substituting x is possible, but more difficult
(A - x²)/4x = h
V = (x²) * (A - x²)/4x
V = (Ax - x³)/4