I may have misinterpreted your post here, but Kaidou's weakness seems to be his drinking habit similar to Big Mom's weakness being psychological/hungry. These weaknesses are not actually enough to be exploited by Luffy.
I believe simply because of both Yonkou's actions. Granted, both of them...