I think the fact that he lived such a long life may be part of it. I don't know when the 2nd Hokage died, but the 1st and 4th didn't have as long of a run as Sarutobi did. There's probably so much he accomplished. Until we get to see him in his prime, just have to believe the story Kishi made...