Look at the manga though. Fight after fight shows characters playing jutsu dodgeball, doing minimal damage - if any. Finally, it comes down to the combatants being low on chakra, and one of them ends up stabbing the other one.
Punches and kicks hurt, and weapons like swords have proven to be more lethal than most jutsus; and the only chakra they require is how much it takes to swing them.
That's because going in for taijutsu attacks is pure suicide against anyone in the top 20-ish. Like if Rock Lee or Ay or Kimimaro went for a strike, they'd be smashed with a chakra arm and blasted with Frog Kumite, flicked away by Susano'o, teleported with FTG, sniped/evaded with Kamui, have a forest created under their fert, face a monsoon of crushing sand, get sucked in by gravity and catch a chakra rod to the head, etc. The way the game has evolved, pure taijutsu isn't exactly a good game play. Even taijutsu experts are creating massive attacks like Gai's Hirudora/Evening Elephant.
Your post does hold merit, a lot of batles do come to simple blow for blow. But, with how the game has changed and power has scaled, a ninjutsu expert would wreck a taijutsu user before they came in range. Like Madara with Bijuu sized Katon, Itachi's wall of Amaterasu, what have you. However, taijutsu and physical damage is desperately needed when all that chakra is gone like you pointed. It definitely shouldn't be neglected in any light.
TL;DR - Ninjutsu generally has a bigger scale, more devastating results, and doesn't require placing yourself in dangerous situations, as opposed to the bulk of taijutsu. However, both are incredibly useful and necessary tools for the standard shinobi.