why would luffy need to fight against a swordsman when he has a swordsman in his crew ?
What about Shanks and fujitora? But essentially my question was fist vs sword at the highest level of both?
Your answer is WB Garp and Roger once again. Roger may have used a sword but I'm not ready to classify him a swordsman, Rayleigh was using a sword at the highest level. Swordsman have no significant advantage against fist fighters in the One Piece verse.
Majority of swordsman (or similar fighting style) are right hand men.
Shilliew
Zoro
Rayleigh
Kaku
Daz Bones
Ohm
Killer
etc.