You don't know if Shanks is a swordsman, there are people who fight with a sword who still don't qualify as a swordsman.
Blackbeard also has used a sword and he has three guns but also two devil fruits and his punches will break your neck and most likely has awesome haki. Is he also a swordsman or is he a gunner, or is he whatever comes in handy for the argument.
In One Piece there is a special breed of people who call themselves swordsman, and take pride in doing that. Zoro, Kin'nemon, Tashigi, Mihawk are all swords(wo)men, but I don't you can qualify people like Law, Blackbeard and Arlong like that.
There is a difference in training and honing your sword skills every day because that is your way of life and having a sword hanging from your belt and whipping it out when you need it.
It's also silly to think that the guy Luffy is trying to surpass is weaker than the guy Zoro wants to beat.
A swordsman is anyone who predominantly and consistently uses a sword. Arlong isn't a swordsman because he doesn't predominantly or consistently use a sword.
Teach isn't a swordsman because he has never been seen using a sword. He uses claws, I'm almost certain you're thinking of Marco, who also isn't a swordsman because he doesn't meet the criteria above. Teach also isn't a gunner because he doesn't use his guns predominantly or consistently.
Shanks is a swordsman because he does use his sword consistently and predominantly. Every instance of Shanks about to battle, he's used his sword. When about to confront Whitebeard, he drew his sword, and this is notable because Whitebeard drew his Bisento AFTER, so it's not like Shanks drew his weapon as a response to Whitbeard using a weapon. When he suspected Ace of wanting to fight, he grabbed his sword. He stopped Sakazuki's fist with his sword, and the first thing he did after declaring battle in the War of the Best was draw his sword. All of these show that Shanks uses his sword predominantly and consistently, thus, he is a swordsman by portrayal so far. Is it possible that Oda may reveal something that shows Shanks doesn't predominantly and consistently? Of course, but we cannot go on guesses on what may happen in the future, only what we have now, and what we have now, is that Shanks is a swordsman.
Law is also a swordsman. Why do you not consider him a swordsman? Because he uses a separate power in conjunction with his blade? Does that mean Zoro and Rayleigh aren't swordsmen because they infuse Haki into their weapons, which are, like Devil Fruit powers, a separate and distinct power from their swordsmanship? If you don't consider Law a swordsman because he uses his sword to conjure his Devil Fruit powers(he also uses it as a normal sword, as he is now against Doflamingo and as he did when dueling Smoker and his Jitte), then you cannot consider Rayleigh or Zoro swordsman because they conjure their Haki into their swords.
Honing your sword every day because that's a way of life doesn't make you a swordsman, it makes you a passionate swordsman. A man that trains his swordsplay once a week or once a month, he's still a swordsman, he just lacks passion.
You are free to not consider Shanks and Law swordsmen, that's fine, but it does not matter to forum discussions, that's your personal feeling. It still stands that, as per portrayal, they are both swordsmen.
Luffy doesn't want to surpass anyone. Luffy has never wanted to get stronger for the sake of surpassing particular people, but he wants to get stronger for the sake of getting stronger so he can be sure to protect his friends, and for the hell of it. Luffy's and Shanks' agreement is to meet together again when Luffy is a great pirate, that has nothing to do with battle. If ¨meet again" had to do with battle, then Shanks could have gone and seen Luffy at Marineford.
That would still put him above Fujitora and Rayleigh then, so Mihawk is above admiral level either way. We saw WB use his Bisento originally, only later on did we get to see his real powers. Shanks could very well be the same considering that was 12 years ago, and in 12 years time, many things can happen, he wasn't a Yonko before, for all we know he was a scrub (who couldn't knock out a sea beast with CoC), yet in that very same 12 year time-frame he became a Yonko. Something must've happened after meeting Rayleigh, who knows.
So be it, Mihawk is stronger than Fujitora and current Rayleigh. There are no problems there.