Would you tell someone that as soon as you meet them?
Because, to be completely honest, I think part of the problem with anime is the bad cosplayers.
If people walked around dressed up as characters from Family Guy, I doubt anyone would want to watch that show either.
So when I think of the word 'anime', what comes to mind is some overweight guy with pimples holding a plastic sword and looking absolutely ridiculous.
Only girls who possibly might think that dating me would be a good idea.
Otherwise, no, or rather, it depends.
For example: If I met someone who wore a t-shirt with a Naruto motif, I'd probably say something like "Oh, you also like anime?" or something.
Other than that I wouldn't really be flaunting with it, but if they asked about hobbies and such, I'd say that watching/reading anime/manga is part of it(part of my life actually).
Aaaah~ I think I get what's bugging you~
Prejudice.
And also, you seem to focus too much on reality here, anime and manga is fiction, I enjoy reading them, not because I saw a hot babe reading manga and I started reading it just to get closer to her, no, I like anime/manga because the stories themselves are entertaining and so much more.
The fans are something different, they're not the manga nor the anime themselves, they're just fellow fans, and you'll be surprised to know that they do quite differ from your preconception, though there are those fitting the picture you paint, even then, so what?
Is Naruto worse now that you know there are people whom enjoy cosplaying, even though they don't always look the part? Now that you know there are people on this forum and other forums whom troll and flame others for whatever wretched reason?
If you are afraid of not being accepted for liking something like anime/manga, afraid of being alone, then I'd like to refer you to your closest anime/manga convent, and confirm with your own eyes what people your fellow fans are: Is everyone an overweight, pimply person(acne isn't really restricted to our lot I'll have you know~)?
Are you really reading Naruto or are you just flipping through the pages, busy doing something else when watching the anime?
Sorry for the essay, I tend to become long-winded with things I like.