The anime just makes him look like an incredible douche that doesn't actually care for the village, but his status.
As for the manga, I still think he's an *******, but he genuinely cares about Konoha and wants to keep it safe at all costs. His methods don't seem like they're the best, but what methods are? Both the way Hiruzen and Danzo run things have gotten people killed, so who is to say Danzo is going about maintaining peace the wrong way? Maybe the sacrifices he makes/wants to make are necessary.