Final answer:
If two masses connected by a string over a frictionless pulley have different weights, the system will accelerate rather than staying in equilibrium, moving with constant velocity only after external forces balance gravity, or without moving at all.
Step-by-step explanation:
If the system consists of two masses tied together by a string over a pulley and there is no friction, it will not be in equilibrium unless both masses are equal. If one mass is larger than the other, the system will accelerate towards the side with the larger mass due to the unbalanced force resulting from the difference in gravitational force on each mass. This means option (a) 'The system is in equilibrium' would be incorrect unless the masses are equal.
Option (b) 'The masses will accelerate indefinitely' is incorrect because the acceleration will not be indefinite; it will approach a constant value determined by the mass difference and the force of gravity. Option (c) 'The masses will move with constant velocity' is only true after the force of gravity is balanced by another force, which is not the case in a frictionless environment. Option (d) 'The pulley will rotate but the masses won't move' is incorrect because if there's a non-zero net force, the masses will move. The correct situation is that the system will accelerate until an external force stops it, reaching a constant speed during free fall.