Wasn't even close to his prime. If you read the manga you would know he got weaker because he gave up on what he truly believed in. Plus Mifune gained knowledge and a sharper "blade" while living. When they were younger and both were in prime hanzou negged him by playing a swordsman game and didn't use ninjutsu at all.And still a Jonon lvl Samurai negged him when he was Edo which is > his prime