I mean look at it like this.
You've read the spin off series so I'm sure you remember All Might rushing from a building and to the falling sky tower in a matter of a seconds. Not too mention that he removed all the people in the surrounding area in less then a second. If All For One can fight...