No I don't think it matters who your family is. But it is kind of ironic how all the Hokages have been related in someway. The first and second were brothers and Hashi probably trained Tobimaru at some point. Then Tobimaru was the leader for Hiruzen's squad. Then Hiruzen was the leader of Tsunade's squad and then Jman trained Minato. And they also considered Danzo, (in Tobimaru's squad) Jman, (trained by Hiruzen) and Kakashi (trained by Minato) for Hokage at one point or another.
So they may not be family but they all where connected through the squads and training and what not.