Seeing how basically no one can touch him, what do you think his weakness is?
Except for the obvious (Seeing Luffy, and fanboying moments)
I say :
His Barriers seem to only work on one side,so if he shields one side,a person can just attack from the other side