Let (x, y, z) be a point on the plane in the first octant. The box formed by this point has volume xyz, and you want to maximize this subject to the equation of the plane.
Use the method of Lagrange multipliers: the Lagrangian is
L(x, y, z) = xyz - λ (x + 2y + 3z - 6)
Find its critical points:
∂L/∂x = yz - λ = 0
∂L/∂y = xz - 2λ = 0
∂L/∂z = xy - 3λ = 0
∂L/∂λ = -(x + 2y + 3z - 6) = 0
Solving the first three equations for λ gives
λ = yz = xz/2 = xy/3
Solve these equations for y and z :
• yz = xz/2 => y = x/2 => 2y = x
• yz = xy/3 => z = x/3 => 3z = x
Substitute these solutions into the last equation and solve for x, then again for y and z :
x + 2y + 3z - 6 = 3x - 6 = 0 => 3x = 6 => x = 2, y = 1, z = 2/3
At this critical point, the maximum volume is
xyz = 2*1*2/3 = 4/3