Well that's the thing, he cares whenever he is around, and he wasn't around for a long time, we don't know the details of the circumstances he was in, but it is logical to assume that if he is in far away lands jumping dimensions tracking beings beyond Kaguya, going for casual family visits in not an easy option.
Anyways, I also think the long absence is a plot hole and just bad writing, because when he did appear he proved that he does love his family, meaning the circumstances he was under did not allow him to visit but Kishi just never explained.
It's a plot device for Gaiden. Kishi wants a dramatic story yet he doesn't think things through and expect readers to question everything to this extent.
That's not even the point anymore. Kishimoto, deliberately or not, has messed up the timeline where Sasuke has been a father before this mission started, and his daughter has no recollection or memories of spending time with him and getting to know him.
It's sad that Kishimoto failed to showcase:
I. Why Sasuke never came home, or sent letters, when he has shown being able to.
II. What Sasuke was actually doing while he was away
Because as it seems, the movie villain comes to attack Naruto/Sasuke fights the subordinate , and then later Sasuke and the Kages/Boruto use the S/T jutsu to go fight Momoshiki and rescue Naruto in another dimension.
It's just lazy writing all together. Remember your idea of dimensions making time slower, and that was the reason why he didn't recognize his daughter?
It would have made for a better excuse than for him to be wandering around dimensions and not having found anything for years, and not getting the idea to come back home, maybe at least once a year if not at best?
C'mon Kishi, that's retarded and readers know it.