Exactly also everything is limited to range of his sight.of course he doesnt need objects to swap with, swapping is just one aspect of his s/t ability. he did the same to kaguya when he ported close to her. he can either shift himself, shift others to him or he can swap with something or someone.
He can not teleport past range of his eye.
Just imagine if he has Byakugan vision along with that left eye... lol teleporting at radius at least 1 km or even further as Hinata's byakugan has wider range of sight.