Nami managed to Lead the Islands towards one of Grand Line's gargantuan Storms, which was the only thing in the way of Shiki's path to the East Blue. That started to destroy his Island and all the work he did. Earlier that day, the Straw Hats had already invaded the palace and opened fire on all his allies. Both of those events took a toll on Shiki, because he spent 20 years arranging that plan, and seeing it being messed with (and eventually stopped) by another man of East Blue, Luffy. Likewise, Luffy used his rubber body do deal a devastating 3rd Gear Kick infused with the storm's lightning.
One can draw the parallelism with previous events, when Shiki confronted Gol D Roger. His plans to rule the world were ruined by one man from East Blue, a great deal due to a Grand Line Storm, even though Roger was clearly outnumbered. Which is to say that when this happened again, now with Luffy, Shiki wasn't in his best mental sanity, which ultimately lead to his defeat.
