All that happened between them was that:::
1. During the Uchiha massacre, Hiruzen was the only one out of the four elders (danzo being one of them) to not agree with that action, but it happened after danzo and the 2 other elders urged itachi to do so.
2. During the Third Shinobi World War, Danzō agreed to help Hanzō of Amegakure in order to receive help in claiming the title of Hokage. He sent some of his forces to deal with the Ame Orphans but they were all wiped out by Nagato.
3. In oder to claim the title of hokage from hiruzen he also established Anbu faction called Root, almost as if he's the appointed "hidden" hokage since the ANBU should only answer to the Hokage appointed.
4. The fourth raikage believed that Hiruzen died because of Danzo plotting and assisting Akatuski and Orochimaru during the attack.
5. Danzo was nowhere to be found during the Nine-tail attack on Konoha and he was also hiding during Pein's attack on Konoha as well, proving that he will try to take anything like a full-on-attack on his village as opportunity to stay hidden till it's safe to come out and take power from whomever is hokage, whether it's Hiruzen or Tsunade...
But never did he ever say to Hiruzen and him fought for the title... all that was said was that Hiruzen BEAT Danzo for the title of Hokage by being appointed a Kage by the first and second during the war. That wasn't an actual fight, just stating that he was appointed the title over him....
ACTUALLY.... When Danzo was killed by Sasuke he said thinking of Hiruzen.. who was I to you? or something along those lines, sounding like he admired Hiruzen more than it seemed.