I played Mario 64.
It's annoying because you swim too slowly and you can drown. I remember a level where you have to dive at the very bottom of the water to get some coins so that a star appears. I think there was also a shark or an octopus there while you were swimming.
In the 8 bit Mario games, it's annoying because you have to dodge a million things coming right at you while constantly pressing the A button to float.