Depends on the Rayleigh in question
Old Rayleigh vs Akainu
Old Rayleigh could hold his own against Kizaru(around Akainu lvl) despite coming out of 20years of inactivity, not holding his sword since a long while(even injuring him in the initial clash when Kizaru wanted to dodge him), but Kizaru was pressuring him, indicating that Kizaru would have won in a long run. I'd say around high difficulty. But current Rayleigh may stronger than 2years prior considering that by training with Luffy, he likely recovered some of his previous skills. However getting older may also put him in difficulties longer the battle, so i'd say the winner(Akainu/Kizaru) will take this around high-extreme difficulty
Prime Rayleigh vs Akainu
Basically a young stronger version of Old Rayleigh that could push Kizaru/Akainu to high-extreme difficulty. He even implied that in his prime he could take on Kizaru, Sentomaru, Pacifistas, Kuma(well his back-ups) and thus implying a difference in level enough for me to think he would take this with around high difficulty
As for Old Garp(presumably) vs Akainu i believe Old Garp has stronger haki but not by much and Akainu may manage to outlast him in the long run. Relying entirely on physical strength and haki in his old age may cause some issues giving Akainu the win with high-extreme difficulty though