Ok, Where should I start...
Genjutsu, literally meaning Illusionary arts are the ninja abilities that unlike Ninjutsu or Taijutsu that deal physical damage. Puts the enemy into an illusion effect.
There are 2 types of Genjutsu in general, Chakra fueled ones and others who aren't. Those who aren't chakra fueled, are simply paradoxes or illusions which can be overcomed with simple logic, while the others are a lot more complex. And are casted on one of 3 different ways. Those being : Doujutsu enabled Genjutsu, Sound based Genjutsu and Normal Genjutsu.
Genjutsu work by disturbing the chakra flow of the enemy with your own chakra into the brain disrupting their senses. During these actions you simply force the senses of the victim to perceive things that simply aren't there. So you can trigger each of the sense's and make them See, feel, smell, hear close to everything you want. But that requires you to manipulate his chakra, so you need to be close enough for the connection to be stable. So the main limitation of Genjutsu is only Range...
Genjutsu work's entirely on breaking the balance of that chakra flow. By disturbing the chakra flow with your own, you can put the enemy into the genjutsu. You simply disturb the spiritual energy with the physical one and dragging the person into an illusion. But the most important think here is that you are gonna need a medium, or something that triggers the genjutsu. That in most cases is a handseal, maybe a sound, light and even eyes contact. This will help us focus the chakra flow inside the user's brain disturbing the chakra flow and dragging the enemy into an illusion.
Genjutsu only manipulate the senses, so whatever happens to the users mind, the body will stand still at all times. Every single attempt to control the real body will mostly cause the enemy to only move in his imagination. But there are some genjutsu which focus only on one specific sense, allowing the information of the other senses to come correctly to the brain. Like for example only altering the vision making someone see a field of flowers while in reality there will be a cliff.
While many think Genjutsu can be casted instantly there will always be a small time frame in the middle of the Trigger and the actual jutsu, and that time frame might be the best time to counter the jutsu. This is the moment you have to be fully focused and the genjutsu might take effect very fast, but if you don't the enemy might get free from it without needing to worry a lot.
Genjutsu Detection is done in 2 basic ways, Detecting it by relying in your personal logic and noticing the fluctuation of the chakra flow. Once you are suspicious about something in particular you must analyse everything carefully and make an educated guess, to see if you are right or not. While it is really a dangerous game it always comes down into a gamble to see if you actually are in the illusion or not. As for the second way, to actually be able to notice the fluctuation, it entirely depends on the skills. If you are bearer of gifted eyes like sharingan or byakugan, you can to some degree realise everything and counter the genjutsu accordingly. Chakra sensors are different, they first need to realise what is wrong in order to check the chakra fluctuations with the Sensory ability.
So after finally noticing you are in genjutsu, the most important job is to release the jutsu and that can be in 3 actual ways.
And they all come down to altering the connection of physical and spiritual energy and stopping the chakra flow for some brief seconds. After that is done the genjutsu can be finally released. One way might be the receptors of pain, despite genjutsu can actually alter them, there will always be a difference between real pain and one that is made up. So by actually hurting yourself, you might be able to snap out of a genjutsu.
Another way, might be doing the fluctuation, stop of chakra flow yourself. And that is basicly done by the will power someone bears. As you realise you are in genjutsu, you can still control the chakra flow around the body, and if your control and will is strong enough you can change the direction or stop that chakra flow from the brief moment which will get you out of the genjutsu. But that entirely depends on the training as well, the more skilled you are, the better chakra flow you have, and the bigger is thw chance you can get out of the genjutsu. And the signature skill that most likely does that, and that is the genjutsu release Kai, a jutsu created for the sole option of getting outside the genjutsu.
And the last way, is the way that is most secure of course, is when you have another chakra, which will be fueled inside you, once that is done, the chakra flow will be altered and unbalanced actually releasing you from the jutsu. So, having a partner giving you his chakra, like fighting with a friend, being a jinchuriki or being in sage mode(natural energy), can automatically cast you out of the genjutsu.
Now the last part of genjutsu if genjutsu layering. It happens when the user, or his allies cast 2 or more genjutsu on a person which don't conflict with one another, causing the illusions to be mixed and even stronger. It works based on the level of the genjutsu. Like if a lower level genjutsu is used over a weaker higher one, it would simply fill the empry space. If the stronger/ higher ranked one is used afterwards.
To counter these kinda of genjutsu, you simply need to release the one which has the highest rank, and you automatically counter both at the same time. In cases when they are at the same rank, you simply have to counter one of the genjutsus and both will be rendered useless directly...
Boy...That was a wallie, even in my standarts >.>