I won't mention what Itachi said when he originally saw Jiraiya with Naruto because we can't be sure of the validity of the comment.
Kisame on the other hand said that Jiraiya would drag the name of the Seven Swordsmen of the Mist through the mud. Unlike Itachi, he has no reason to theoretically pretend to think he would lose to Jiraiya. Kisame is a smarter ninja than people give him credit, and if he would make such a strong comment he clearly doubts his own chances of winning.
Beyond that, Kisame's only element is water. Jiraiya uses fire AND earth. Water element is weak against earth. Jiraiya has the type advantage here. Lastly, if Jiraiya summons Gamabunta, he would be one of the few people who would be able to escape from Kisame's giant water dome. Just one jump and Gamanunta would be out of there.