He actually never said that. In the most accurate translation, which is Viz, he said "You've taken heavy damage already."
I was only including damage not inflicted by Doffy himself, as we're looking at external factors that could be hindering Doffy and Luffy in the fight against each other. Black Knight is part of Doffy's arsenal, and doesn't count as an external damaging factor to Luffy vs Doflamingo. And Doffy is only inferior to G4 right now because of the damage he took reducing his capabilities, which is probably most evident in the fact that Doffy currently couldn't notice a Jet attack until it was right at his chest, while an inferior Black Knight earlier could swat a Jet Pistol aside like it was nothing. Luffy hasn't received nearly as much damage as getting your organs torn apart.
Manga indicates otherwise. Jozu is portrayed as an immensely serious character, as he's been depicted smiling. By contrast, many of his peers and nakama were shown smiling at some point or another in the war, so him being in a war isn't an excuse. His friends were fighting all around him, Ace was in danger, and Whitebeard was being fought by Admirals. It makes no sense for why Jozu, a character depicted so far to be extremely serious, would let himself remain trapped by an enemy under these circumstances.