There are a lot of people that could probably defeat Danzo. Especially if he gives them time to figure out his abilities, like he did in his battle with Sasuke.
Overall, Danzo was weak. He relied heavily on powers taken from other people, and failed to fully learn how to use those abilities properly. This left him vulnerable in his fight with Sasuke for two reasons. First, he overestimated his abilities. He assumed that since he had Hashirama's cells and 11 Sharingan, he was pretty much untouchable. Secondly, he did not have full mastery of his borrowed abilities.
I think that you are forgetting that Danzo could have used KA on Sasuke during their battle. Granted, it wasn't until very late in their conflict, but he still regained the ability before he died. However, with KA, like every other Genjutsu, it still runs the risk of being discovered and dispelled. In a solo battle, this means that Jinchuriki are unaffected by KA, and others can use their summons to counteract genjutsu. Additionally, Danzo can't seem to keep track of time in his head. Instead, he had to continuously check his arm while he was using Izanagi. Furthermore, his Hashirama cell transplant seems to be highly unstable, and he needs to suppress it with his chakra.
All of these factors contribute to him not being as capable as some people give him credit for. In other words, his abilities would be enough to defeat any fodder ninja, or even some higher level named characters, but anybody with particularly good analytical abilities or unique traits should be able to defeat him.