Yeah it wouldn't be incest considering Hagoromo was 1000 years before the time of Hashirama and Mito.It's not incest. It was already known that they were distantly related. Probably 8th-18th cousins once or twice removed. That is not close to nicest. So many generations passed that ll this was completely forgotten and thought to be legend.
I might I'm Princess Diana is my 8th cousin once removed. Genealogy my friend. It exists
But this would explain as to why the Senju and Uzumaki are "distantly related''!