No I don't, he may be underrating everyone considering Luffy, Zoro, Sabo and Fuji as his potential problems. I think he's gonna have some kind of scheme to either take out the Marines including Fujitora or he will make it to where they can't interfere with the family. He'll probably do the same with the other people Doffy is after. Either way, I don't think he's gonna overpower Fuji to get an upper hand if he happens to get an advantage over them somehow.