I believe he was out doing stuff, I think he learned of his death after the fact, and the village generally decided not to talk about Naruto because of the Kyubi.
Being not there for him doesn't mean that he didn't took care of the kid.
He was just there, gathering info on the Akatsuki, also research for its "Make out Paradise", so on and so forth.
When he saw the danger was gettin even bigger, he just returned to protect the kid.
Until then, he entrusted Naruto to the Third Hokage. It's not like he left him alone for good. He knew Sandaime would take good care of him and he didn't need to worry.