He killed them all before he needed to use it. Him dying by their hands is a nasty rumor the cloud village made up out of jealousy. Really Tobes just retreated to his island getaway after he made hiruzen hokage. He just didn't dispell the rumors because he liked his life in paradise better than his duties in konoha.