Ok before you all go crazy think about this. When sasuke had that first conversation with sakura he called her annoying. Thats because what she said to him got him angry but instead of showing it by lashing out he called her annoying. The next time he did that was when he was leaving the village and she pleaded for him to stay. Instead of reacting to this he called her annoying and knocked her out so she would not follow. Once again we see sakura pleading to sasuke which affects him so he calls her annoying and knocks her out so she wont follow. He calls her annoying because it annoys him that her words can get to him like no one else.