The fact that nothing notable about him has been said ever by any character. Other theories in which the elder son and Izuna are Tobi have more textual evidence supporting them. Hell, Izuna could've been revived by Madara once he obtained the Rinnegan. It's more possible than a character we've heard nothing about other being Tobi.
Here's a theory: Izuna was revived by Madara after he obtained the Rinnegan. The brothers formulate a plan to get vengeance. At some point, Madara dies, leaving Izuna to follow through with the plan on his own, adopting his brother's name due to the amount of fame. To combat age, injury, or perhaps to substitute a part of his body that decayed before Madara could revive him, Izuna replaces parts of his body with Zetsu goo. Izuna has the motive to be Tobi, he has the background and knowledge of history to back up what Tobi has said, and, although it's probably more coincidental, Tobi is listed as 175cm, 59.9kg while Izuna is listed at 174.8cm and 59.9kg.
Izuna fits, more so than Kagami. I'm not implying that Tobi is Izuna, I just think it would be more likely than Kagami. To suddenly reveal that Kagami is Tobi would seem like a huge ass-pull on Kishi's part.