probably with his father
it has been said shisui inherited his father's will which was devotion to peace above all else and of course this + itachi's pacifist nature turned itachi into who he was.
When Hiruzen smiled and said he reminded him of Kagami, he did look quite respectful which indicates he held no ill bearing and probably like a son would loved his dad and is the most likely cause
at the end of the day, doesnt matter