Well, considering that the entire Uchiha clan was plotting to take over Konoha and kill who knows how many innocents on the way, we just haven't been harsh enough. I take Itachi and Shishui were the exceptions and ones of the only Uchiha's ever (obito along with them) to be decent people.
Now, seriously, what most people fail to see is that while the Senju (and Konoha along) reward and admire integrity and decency, the Uchiha clan admires and rewards strenght and power. That's the basic distinction between the 2 brothers as well. For those of us (readers) who believe in the first line, we will always see the second one as flawed and "evil" or at the very least wrong. For those would believe power is the mean for anything (AKA the older brother), they will always defend this line of thinking, for it is what they believe in.
This is a matter of beliefs, not the manga itself, but who we really are, therefore, we will stand up for it, each in its own way.
Now, back on the OP, I do believe the Uchiha would be just a normal clan without the Sharigan, maybe not as weak as some of the clans around, but surely not one of the most powerful clans ever. Yet, we must keep in mind that even without it, some ninjas such as ITachi would still be a genius, and therefore a very powerful ninja, and would find other ways to enhance himself.