Here is my theory, which combines some of what I've read, so i can't really call it my own personal theory, but it's that he's holding off Sabo to give an excuse as to why he wasn't fighting the SHs, because he was too busy fighting Sabo to fight the SHs, so he's actually trying to not kill him here. And that also ties into the other marines, that they can't fight the SHs because they're caught within the fight and can't reach the SHs, which refers to the panel about how they can't reach the SHs with all the commotion, though mangapanda has it translated really weird. It's the first of 2 pages of their clash.
And it applies to what Fuji was saying about the game and risking his life in that he's betting that Luffy & Co. can defeat DD. He know the SHs mean well, but he doesn't want to get involved personally where he can lose his position since people will find out unless he decides to kill everyone to keep it quiet, but that's clearly something he won't do. So he's relying on Luffy & Co. to defeat DD for him, so he can keep his position, while at the same time can give the excuse he was too busy fighting with Sabo to intervene. But if they lose, he's in a really shitty situation, because then DD would annihilate everyone and probably escape and he'd lose out entirely. I think that Fuji feels like if he loses his position or if DD gets away with this, then his whole plan will fail and he shamed himself possibly to the point of suicide, as you know Japanese feeling of shame and dishonor and such is very different from the Western view.
That's my theory. And if you're too lazy to read it, tl;dr, I think Fuji's definitely holding back and actually not trying to kill him.