A Genjutsu is a technique that manipulates your senses, you need to have a strong awareness of reality in order to understand when something is out of place.
It's very crucial that whatever reasoning you do, you do it fast. Sometimes I see people go into long discourses about some complex realization that their in a genjutsu, when they could have just focused on a small detail and go on with it. Obviously, the easiest way to figure out you're in a genjutsu is about how bizarre a certain situation is, either because it's too fast or unrealistic, or because you can't pinpoint how it began, or even because it's working in a way it isn't supposed to. Something like the sky suddenly turning red, or things starting to move around or distort, usually means a genjutsu.
That is the second way of realizing you're in a genjutsu, when you know the ninja or know enough to understand that whatever the genjutsu is creating is beyond the capability of your opponent. For example, if you see a byakugan user create a tree. You have enough understanding of RP to know that these KG don't match. This is also the case of people create genjutsus of animals that they cannot summon. Except Pain, a ninja can only summon one animal type, so if someone you don't know summons an animal, and then creates a genjutsu of a different animal, you can pretty much assume you're in a genjutsu.
Of course, if you do not know the ninja you're fighting, you have to assume some hidden skills, and learn what is real and what is not by trial and error.
Other than Bizarre Effects or Unattained Skills, you need to rely on the Genjutsu's own limitation. Especially in lower ranks, a genjutsu will only alter a couple of basic senses. Sometimes, a genjutsu can be purely visual, so, sounds associated with what the illusion is creating won't be produced, and you can use that to your advantage. Other times, you can see some light changes or superposition of images (for example, a person might see an illusionary animal or clone that doesn't bend grass when it walks). You can see a tree constricting you but not hear it grow. etc etc. When a genjutsu breaks the B rank mark, it starts to complexify, especially in the hands of a good genjutsu user, and start to describe a richer experience, but even masters make mistakes, and even powerful genjutsus have limitations, people just need to explore them.