Why does most people want him to be evil
I think quite a few people just have the instinctive insight to see that something is off about Hagoromo. It's not just 'how he looks' - it's also in his character and the events surrounding his decisions.
Jiraiya is a largely loved character. While he may not be involved much in the "my favorite character is stronger than your favorite character" bantering - his character is very much in line with what one would expect of a "sage." He knows when to be serious but is often just rolling with whatever is in front of him. His character isn't complex or intellectually intimidating - but he's no one's fool, either. His strength comes not just from his own ability, but in the way he behaves and interacts with the world around him.
Jiraiya is the closest thing to a Sage the series has seen - spare perhaps for Naruto, but he is not really there just yet (the series will end with him understanding the true nature of a Sage and embracing it - he is currently toeing the line Hagoromo crossed back when he first sealed Kaguya). Which is what the upcoming conflict is about.
The "Sage of Six Paths" is not 'evil' in the western sense of "I want to do bad." He is 'evil' in the Eastern sense of "I will stop at nothing to do good."
To see how to epically fail at this concept - watch Star Wars: Episode 3.
The Sage of Six Paths is essentially Darth Vader (except Lucas apparently didn't write his own story... or something...) - a man with immense power who could not stand to see the world around him behave so unjustly. Embracing the idea that those with power should always use it for the better of the world - he began justifying everything he was doing with "for great justice (take off every ZiG)." Once you begin treating others as 'pawns' or 'tools' within your view of justice - you have effectively become evil because your actions are taken out of a purely selfish view of how the world -should- be according to your ideals as opposed to accepting the world as it is and learning how to live within it.
Lucas kinda-sorta tried to do this with Episode 3 by showing Anikin worrying over Padame's death during childbirth and the whole conversation with Palpatine about 'stopping death' - but it really failed at illustrating the concept (though I understand what Lucas was trying to do - he just doesn't know how to tell a story). Instead of being 'seduced' by 'the dark side' - Vader seems to have been 'tricked' by Palpatine into being evil (which... doesn't really make sense - it's like tricking someone into being homosexual).
Cue the "Aim said homosexuals are evil" crowd.
You laugh and roll your eyes - but if my stuff was read at the national level, you can bet Salon would have a featured article about how I insinuated homosexuality was evil and it's a horrible stereotype, oppression, etc.
Anyway - the point is that Hagoromo has embraced the idea of "the ends justify the means."
People pick up on this mentality through his actions - even if they don't consciously understand why. They get the impression that people are more like objects or tools in a greater plan/scheme. They will follow him because he has a plan and an idea - but they are nervous around him because they realize:
1) The end goal is not their own and not made known.
2) Tools are replaceable and many of them disposable.
This is why many are suspect of the Sage of Six Paths.
Though I have been speculating that he is 'evil' for quite some time:
You must be registered for see links
This was an earlier theory regarding why we do not see the Hyuuga utilize any form of ninjutsu even when it would be practical to do so (substitution, transformation... even a simple clone for diversion to close the gap).
Basically - the Hyuuga reject the use of Chakra and/or Ninjutsu as a heretical practice that goes against the natural course.
Although if they draw their ancestry back to the Ototsuki - it's possible that they physically can't use ninjutsu in the first place.
Although I speculated the sage was 'evil' almost a year ago:
You must be registered for see links