The moment he unlocked kamui he became low Kage. The fact that you can't hit your opponent makes it difficult to beat him. Also not many s/t users were around at that time who could take on kamui. So I say when rin died.
If not then than it's when Madara taught him about yin and yang. However long that took.