We all know he's an incredible swordsman on par with Mihawk and that he has powerful Haki.
While it's true that he is powerful haki user and it's very likely that he is incredible swordsman, not once it was stated that he is on par with Mihawk. Mihawk is best swordsman in the world.
Do you think he has?
Would you prefer him to have one or no?
No. Like someone already pointed out he made clear in first chapter.
Majority of the strongest characters are devil fruit users, it would be nice to see some others to not be one and instead relly on haki and their own strength skills.
I've read it somewhere...
That Mihawk never won any of their duels and Shanks never intended to take on the title of the Greatest Swordsman and when Shanks returned w/o an arm, Mihawk was annoyed for the fact that defeating him now wouldn't satisfy him because of the handicap.
Nope, never stated or implied. Maybe that was some theory, but it would be horrible IMO.
Zoro's goal is too surpass best swordsman in the world aka Mihawk, if Mihawk got his title only because Shanks lost his arm and earned it because Shanks got huge handicap, that would be bleh and without a point.
Also there are 4 Emperors: Big Mom (Paramecia), BB (originally Logia) and Kaido (most likely Zoan) so it would be nice if 4th would not relly on Devil Fruit.