You're right. But the thing is, even when he fights, if he loses, somehow he wins as well. This might sound contradictory, but here's what I mean: for example, sixteen years ago, Madara extracted the fox from Kushina, unleashed it on Konoha, fought the Fourth Hokage, Minato, lost in their "battle of speed," as it is called, but what happened at the very end of things? Both Minato and Kushina, along with a good number of Konoha's ninja, lost their lives that night. I'd say Konoha took the bigger blow that night, but it's up for debate.
He somehow manages to remove the most powerful of enemies from the equation, even if it must take him a long time to do so. Another example - Itachi. He was the one who stood in his way when Madara tried to rekindle the flames of war, but he ended up dying, and it wasn't even at Madara's hand. Now he's using Sasuke to systematically eliminate any possible threat to Akatsuki, like Danzo.
While Madara claims to be nothing but an empty shell of his former self, it seems he has the psychological advantage of being able to pull some sensitive emotional strings against his enemies, and consequently using it to his advantage. My theory, which may not be true (of course), is that the First Hokage, who defeated Madara at the Valley of the End, was killed by Madara (disguised as a Kumogakure ninja, possibly) during the First Shinobi War, a way of getting even with his with the one person he admired, and "hated the most."
He only physically fights when he needs to, I agree, but I say that his power comes not from battling Fu and Torune, among other people, but that of his cunning planning. Even Danzo, who operates "underneath" Konoha without the Hokage's knowledge or permission, calls Madara a "sneaky bastard." That to me indicates Madara has a skill for manipulation unlike any other.