The fact that Jozu wasn't paying attention is actually against him. He is in a war, he HAS to pay attention. This isn't an arguement. Doflamingo was playing with him in a middle of a war, where at the same time his captain and nakama are in danger. Plus, he didn't just stop an attack, he immobilized his whole movements completely.However, if you wanna use that arguement, I might as well say that the only hit Jozu managed to land on an admiral was due to the admiral not paying attention. The fight was also off-panel if I remember correctly, so we can't assume that they were fighting on equal grounds. Aokiji basically one shotted Jozu when he found the opening, and cost him is arm.
Plus, Aokiji might have effortlessly attacked Joker, but Joker wasn't even trying as well. In few words, neither were trying. Do you think that if Doflamingo was serious, Aokiji would have frozen him as easily ?