He hasn't, he was able to bypass Fuji's gravity. When he went after doflamingo I'm sure he was stopped by fujitora for plot device.
He absolutely no diff fodder stomped Neptune and all of his people by himself, literally had them all tied up. And I'm positive Neptune is powerful. Stronger than...