You make out like the Sharingan gives him some sort of godly strength when it doesn't. Even without the Sharingan, Kakashi's strength stays the same. To make this absolutely clear, "As the Sharingan evolves, gaining more tomoe, this ability extends to being able to track fast-moving objects before finally giving some amount of predictive capabilities to the user, allowing them to see the image of an attacker's next move from the slightest muscle tension in their body and counter-attack or dodge without any wasted movement."
So, no, Kakashi will not be the weakest hokage ever. Kakashi has a genius intellect, much like Shikamaru. Intellect aside, he's proven himself to be a well-balanced fighter. His Strength and Speed are above average but his Stamina is at the average mark. As proven above, the Sharingan does not give him some godly strength that he would suddenly lose without it. It's more to the point that against opponents with Dojutsu, he really can't do anything due to him being unable to A - Fight them effectively and B - Come up with and use effective strategies against them. This doesn't mean he's weak, this just means that he's handicapped without the Sharingan when faced with Dojutsu users.