The point is there has been no mention of "pure swordsmen" in One Piece, it is a term and concept completely fabricated by this fandom. Oda wrote of Kaku being a swordsman multiple times despite using Rokushiki and his Devil Fruit just as much, and there was no correction. Kaku, nor Zoro, nor anyone period ever alluded to Kaku being unable to be called a swordsman, so that means this whole "pure swordsman" thing is bullshit.
DF can be a deciding factor in a fight, but not for Mihawk. He's the world's strongest swordsman period. That means whether you're a swordsman who uses nothing but a sword, a sword and Haki, a sword and Devil Fruit, a sword with Haki and a Devil Fruit, a sword with a martial art, or a sword with all of the above, Mihawk is better than you because he's just that damn good. End of story.