I just think it was the 1st to make him lose his powers because:
-he says that the fight with the first left him "scars"
-second, in the 500 chapter, when he appears 16 years ago, he already has a mask that shows only one eye
-if he was in full control of his powers 16 years ago when he appeared before minato and kushina, he would just kill minato, subdued kushina and realeased the fox. He was on pair with the first as one of the top 3 caracters to walk the naruto world, the other two being the 1st and the sage. So when in control of his abilities he would not be so "sneaky" in the shadows. He never was in the past.
The "missing" eye of madara may be the key. If the first damaged or rendered his left eye unless, his abilities should be reduced to half at least. Imagine, if Itachi had only the right sharingan he could only use amaterasu, not tsukoyni or susano (that requires both eyes). The same goes for madara. If controling the fox requires both eyes, with only one he could not do it the same way. That's what I believe happened to Madara: The first took his left eye out (or damaged it beyond salvation), making lose (at least) half of his abilities.
What do you think? (I have a conspiracy mind...confusion and wild ideas I know...)