Most peoples' definition of speed is just getting from point A to point B in the quickest amount of time possible instead of physical movement from one place to another. This is somewhat validated in that Minato is able to travel to point A to point B in no time at all, but it is generally incorrect in that the method of transportation that Minato uses doesn't validate with the actual formula used to determine speed, which is distance/time simply because he can reduce the time variable to 0, making the formula undefined and, therefore, the technique FTG, not quantifiable with speed.
In other words, FTG = fastest mode of transportation, so Minato is technically the fastest in the world in terms of transporting himself from point A to point B
but
FTG =/= speed, so Minato is not the fastest person when it comes to physical speed according to its definition.