How can he lost most of it when they only gave him like 5-10% of their chakra? They give him access to their abilities via chakra and he uses Kurama for the actual chakra. Hagoromo and Sasuke both stated Naruto housed all 9 Bijuus. Also Bijuus don't lose chakra unless they split their consciousness with their chakra as they did with Naruto.
Say they give Naruto 10% of their chakra each, Naruto will never waste it since they'll just recharge their chakra as we seen Yang Kurama do so many times in war arc. Rikudo Mode (RM) is an ability Naruto gained after receiving Hagoromo's Yang chakra; the Ashura chakra in Naruto changed to Yang Hagoromo as Sasuke's is Yin Hagoromo. If Naruto lose the chakra then that means Sasuke did too and that means he doesn't have the Rinnegan since Hagoromo chakra awakens the Rinnegan. Ashura (Yang) and Indra (Yin) awakens Hagoromo's chakra (Yin Yang) but instead of awakening Hagoromo chakra this way like Madara, Sasuke and Naruto were given the power directly. And since we know Sasuke still has the Rinnegan, same is for Naruto since it makes no sense for one to lose the same thing (Hagoromo chakra) and the other doesn't.