The condition probably called nystagmus. Nystagmus is a disease that will cause one of the eyeball movement is disrupted.
The cranial nerves that have the most role in eyeball movement are oculomotor (cranial III). But trochlear(cranial IV) and abducens(cranial VI) nerve have some role too. Without information about the specific direction of eye movement, you cant specify which of three nerve that got damaged.