They left when Madara started his crazy things.
Madara battled and battled, cut some mountains, cancelled the contract with Kabuto, talked with his typical pimp style, stetched a little, left the Kages on the brink of death , left and arrived to where the rookies should be ages ago.
But we all...