Use the First Derivative Test to find the value of h that maximizes V(h).
V(h) = h(h - 10)(h - 8)
V(h) = h^3 - 18h^2 + 80h
V'(h) = 3h^2 - 36h + 80
0 = 3h^2 - 36h + 80
The Quadratic Formula tells you that your roots are:
h = 2.9449495367
--or--
h = 9.0550504633
So those are your critical points. Both of those values are within the domain of h, so now we turn to the Second Derivative Test to find out which one is a maximum.
V''(h) = 6h - 36
V''(2.9449495367) = -18.33
V''(9.0550504633) = 18.33
A local maximum occurs where the second derivative is less than zero (and likewise, a minimum occurs where the 2nd derivative is greater than zero). Hence, the maximum volume occurs at h=2.9449495367. Sticking that into V(h) gives:
V(2.9449495367) = 105.0276
The answer then rounds to 105ft³