Kakashi may be kage level material with his experience, large arsenal of jutsus, intellect, and sharingan but with a powerful duo facing him, it's not gonna be easy for him to defeat so easily.
He may be a genius, but Neji is also considered a genius and was the only one promoted to jounin after the timeskip. He also had the byakugan which is very important since with it he can use it to determine when he will be using any jutsus and see where kakashi is at all times. His powerful eight trigram attacks will be tough to avoid, and the gentle fist style is difficult to counter.
Lee on the other hand has pure brute strength and stamina. He is known to be able to open at least five gates, making him very quick and powerful. Although it is unknown whether kakashi can open any more than the first gate, it is very unlikely he unlocked more than that since only ninja who train exclusively in taijutsu are able to open multiple gates. Also, it would drain him very quickly since he has relatively small chakra reserves.
