I assume the answer this question is looking for would be a large amplitude.
Small amplitude and low frequency do not make sense as if the amplitude is smaller it would has less effect on the boat, this logic is the same with a low frequency.
A high frequency would cause more waves to effect the boat which could effect the boat, but it could be a wave with a small amplitude (a smaller wave) that may not effect the boat as much. So if one million waves with microscopic amplitudes interacted with the boat you wouldnt really feel the effect which is why the frequency doesnt matter here.
Now, if you had a larger amplitude wave, the boat could tip. Imagine if one really big ocean wave hit a boat. If that wave hit the boat it is possible it could tip. Therefore, amplitude appears to be the most detrimental for boaters.