You are forgetting sir that Hagoromo spread his teachings and religion: Ninshuu with other people, it would be absurd for him to teach Ninjutsu to other people when only he himself can wield it, logically, aside from teaching the principles of Chakra, he also gave them Chakra to wield through some mystical means (most probably like how Naruto shared Chakra with the Alliance) . And it is another great falsity that people assume that Hagoromo was the blood ancestor of all Shinobi for this is simply not the case, if the only way for the succeeding generations to acquire Chakra was to be blood related to him, then he need not have spread his teachings to other people, he would've just mated with several women to conceive chakra-infused babies. No, the other present day Shinobi Clans like the Akimichis, Inuzuka, Aburames etc. descended from the Pre-Ninjutsu people/families to whom Hagoromo spread the Teachings of Ninshuu and gave Chakra to. And it is only logical, that aside from people not blood related to him, that Hagoromo would also give Chakra to his own kinsmen/blood relatives: The Ootsutsukis. And aside from that, gave them the secrets of his specialty: Fuuinjutsu, the Fuuinjutsu of the Uzumaki Clan. The Uzumaki and Ootsutsuki bloodline being one and the same.
