or the castle serves as a gateway to connect all the dimensions, like when you walk up to the castle from the outside you see 1 entrance but once inside the castle there are many rooms with exits to the outside, and each exit connects to a different world. then kaguya's dimensional travel could be explained as just travel through space but with the condouit always going through the castle. this might also explain why sasuke's summooning worked, he was able to reach his hawk through the castle, just didn't know it.
and the castle of the naruto world would be on some far away location, yet to be discoverd location.
and I think we have seen part of this castle already, I'm not sure what chapter it was but it was this image
You must be registered for see images
here we see the ledge and kaguya standing over it, doing something like what the ten tails did a while back.
I think she is someone who discoverd the secret behind the castle and is trying to concer all of them.