The claim that having greater ocular power as in rinnegan>sharingan makes you immune to visual genjutsu was actually already disproven now that I think about it. Zetsu stated that Sasuke w/ only 3-tomoe sharingan could attempt and possibly genjutsu Itachi and his Mangekyo sharingan, even though it would be unlikely. Shisui had the most powerful genjutsu Kotoamatsukami (which Itachi was later in possession of) and Itachi was the best genjutsu user in the manga. Frog Song is below Kotoamatsukami, considering that Kotoamatsukami was stated to be the ultimate one, because the victim doesn't even know that it took effect on them. Also, Tsukiyomi seems to be just as powerful, the only way it is under Kotoamatsukami in terms of ability is that obviously the victim knows that they are in a genjutsu. So yes, Nagato is affected by Frog Song, Tsukiyomi, and Kotoamatsukami. If he is caught in the latter two, it wouldn't end well for him.