You're right. Anyone with good CoA and a great fighting prowess can beat him.
Furthermore, people dont understand why Law "easily" beat Vergo. Cause Vergo always thought he'd win and he just walked right up to law without going in a serious fighting pose and actually have full-intent on going all-out whereas Law did, and hence he had the upper hand which lead to his victory.