To me Hiruzen was simply a poor leader. He never had the courage to put his foot down when needing to. He may have been seen, known, and had the title of hokage, but Danzo was the one really leading the village.
I mean in the recent chapter when the people of the orphanage said they would go...