Check the picture below.
as you can see, the graph of the volume function comes from below goes up up up, reaches a U-turn then goes down down, U-turns again then back up to infinity.
the maximum is reached at the close up you see in the picture on the right-side.
Why we don't use a higher value from the graph since it's going to infinity?
well, "x" is constrained by the lengths of the box, specifically by the length of the smaller side, namely 5 - 2x, so whatever "x" is, it can't never zero out the smaller side, and that'd happen when x = 2.5, how so? well 5 - 2(2.5) = 0, so "x" whatever value is may be, must be less than 2.5, but more than 0, and within those constraints the maximum you see in the picture is obtained.