The point from a given height to the horizon form a line that is tangent to the curvature of the earth.
The angle of the sector from observation height to horizon point is:
cosa=r/(r+h), where h is the height above the surface of the earth and r is the radius of the earth...
a=arccost(r/(r+h))
And the arc length is just 2pra/360=pra/180
So the distance to the horizon along the curvature of the earth is:
(pr/180)arcos(r/(r+h))
And we have two of these arcs one from the periscope height to the horizon and one from the top of the ship to the horizon.
If we simplify the radius of the earth 3959mi you get
For the periscope:
(3959p/180)arcos(3959/(3959+h)) remember that 5 ft is 5/5280 mi
2.74 mi (to nearest hundredth of a mile)
And for the top of the ship:
3959p/180)arcos(3959/(3959+h)) where h is 50/5280
8.68 mi (to nearest hundredth of a mile)
So the total distance along the curvature of the earth between when the periscope can just see the top of the ship is:
2.74+8.68=11.42 miles