From my observation, the most disturbing characteristic of sociopaths is their ability to be charming. A person who can charm anyone appears as a "red flag" to me.
However, not every charming person is a sociopath, of course. And none among us is incapable of very hurtful/hateful behaviors.
So let me ask you: do you know the differences between a sociopath and a "normal" person?
Intelligence.
Any intelligent person who is also able to charm a wide variety of people is of the same mental breed as a sociopath. The question is merely how they prey upon others.
People like myself do not relish in physical harm or abuse (unless we find the individual particularly revolting); but are going to be quite manipulative in social and mental arenas. That doesn't mean that we are out to -harm- people, or that we are necessarily going to benefit from the exchange - but that we are going to deliberately attempt to engineer the behavior of others in the manner we choose.
That is, in essence, sociopathic behavior; even if it is generally benign.
The people who can charm a wide variety of people without the above are, generally speaking, adorable dolts that others do not feel threatened by.
That's not necessarily to say that 'normal' people are exceptionally different (they will manipulate where they can) - but those with higher intelligence and/or social savvy will be far more successful across the general population. Thus, it will become a more central part of their behavior where others use it more selectively.