Keep it and just defend yourself when people try and make that claim. This site is full of people who have a terrible sense of judgment though, I've been called a Pein fanboy at least 3 or 4 times now.
Come to think of it the term "fanboy" was probably made by someone who was too insecure to have a favorite character and project it themselves. That, or some old guy in a basement.
Doesn't matter if you like Sasuke or what other people say about it. Do whatever makes you happy & take the term "fanboy" out of your head.