Exactly! I mean, it makes no sense why he would hate a character who rarely makes any appearances in the series! Any reason for hating the guy would be a completely shallow reason if you ask me. He must be confusing hatred with indifference, I don't see how it can be anything other than that.
On topic, another possibility is that it could be his student...