I would wager that Hagoromo is actually the Younger of the two.
If we are to take the assumption that Uzumaki and Hyuga come from this other brother, it is likely this new son is older. The Hyuga are known as Konoha's oldest clan. This was then disputed by both the Uchiha and Senju founding the village, and then the Akimichi and Sarutobi both joining before the Hyuga. Maybe even the Shimura I can't remember.
Thus, we must take the words not in reference to Konoha history, but clan history (a literal connotation). Meaning they are the oldest clan that resides I'm that area. Between this and the Hyuga relation to the Uchiha, they'd have to form first... Thus, they would come from Hagoromo's brother who would likely be older to form an older clan.
But yeah the Byakugan comes either from the Ootsutsuki, or specifically the Brother we don't know.