I'd say Minato for the fact that a raging Kyuubi was in the village along with the supposed Madara controlling him.
That was a one time thing, and that is not really a fault that reflects on Minato or his skills at keeping the village safe. Just because a terrorist attack happens doesn't mean that it is the government leader's fault. There was absolutely no way that Minato could have predicted that "Madara" (Obito) would suddenly show up while Kushina was giving birth, and unleash Kurama upon the village.
Besides, by your logic, every hokage struggled with keeping the village safe.
Hashirama/Tobirama's reigns saw the first shinobi world war (in which both died).
Hiruzen's reign saw the 2nd and 3rd war, a genocide, Orochimaru's invasion (in which Hiruzen died), the attempted kidnapping of two little girls (Kushina and Hinata) one of which led to the death of the brother of a clan leader (Hizashi), etc...
Minato's reign saw Obito/Kurama's attack.
Tsunade's reign saw Pain's attack and the 4th great shinobi war (which means that it also saw the juubi, Kaguya and infinite tsukuyomi)
Kakashi's reign nearly saw the world get destroyed by the moon due to Toneri's actions.
Naruto's reign saw Momoshiki's invasion and later it saw the as of yet unknown circumstances behind Konoha's destruction and Boruto vs Kawaki
Tsunade, also this is what happens when you give power to woman .
You must be registered for see images
Sexist, and see above
OT: I'd say Hiruzen. Hiruzen allowed a whole genocide to take place! People like to get on Minato or Tsunade because an attack happened while they were hokage, but neither of those two could have seen said attacks coming or control the fact that they came. Meanwhile, Hiruzen could have prevented the Uchiha massacre. The Uchiha did not have to die. That is why I say that Hiruzen was the worst in that aspect.