He is very brash and arrogant, but he is the raikage and he is known throughout the NV as the fastest man alive through his feats. If anyone else were to become recognized as being even faster than the raikage and has demonstrated that several times in comparison to the raikage himself, then the person will be known throughout the NV to be faster than the raikage.
As the readers, we have a greater perspective than anyone else in the NV so we can infer from feats demonstrated if a person is around the raikages level in speed or if they are faster depending on if we know that they were using their speed to its fullest.
For example, we the readers know that because Naruto was able to avoid the raikages full speed punch, something that only two people have ever done before, we know that Naruto in KM is around the raikages speed. Because we know that BM gives naruto access to both kuramas and his own chakra, it is likely that naruto is considerably faster in BM than KM, though probably not by much since Naruto is only melding his chakra with kuramas to strengthen the potency of their chakra mix to the maximum instead of Naruto solely using Kuramas chakra.
It's different for Gai and Lee because they have not faced the raikage to compare their speed, nor are they well known as the fastest, but us readers know that they have access to the 8 gates, something that most of the NV doesn't know. Also, the two only use the 8 gates as a last resort, so it is something that is rarely seen by anyone in the NV. The most that anyone would know of them is that Gai is a master taijutsu user and probably has a pupil who also is mastering the art as well.
From what we readers have seen concerning their speed feats, Lee in the 5th gate is very likely one of the fastest in the Leaf, second only to his master, who is known in the databooks as being slightly faster than Lee in base and having unlocked more gates than him as well.
If we were to compare their feats and hype to the raikage, we could say that Lee may not be as fast in the 5th gate as the raikage, but he is probably pretty close to it considering that feat was done three years earlier and lee has likely improved much more so on his speed. Since we have only seen base level footspeed from Gai, we can infer from being Lees master that he is much faster than Lee himself in base and has unlocked more gates, and is likely on par and/or possibly faster than the raikage in speed in the 5/6/7th gate. Since we have yet to see the 8th gate, and since the 8th gate has the hype of granting power greater than that of a kage, and the gates grant the user speed and strength, it is almost certain that if either guy or lee were to enter the 8th gate, that they would indeed become the fastest shinobi in the NV for however long their bodies will last.
So with that in mind, it is only because the raikage has not faced guy or lee nor has he heard of them because of how rarely they both use the gates. It is only from our perception that we can infer who is faster based on feats and hype.
