First, you need to know how it works. Second... there is no second step.
You can learn that by tinkering it, looking at how it behaves when you do stuff with it etc. You will probably break the software a few times, but I think it's worth it. Just try not to infect it, unless you make it your goal to understand malware.
There's a huge amount of material and no perfect starting point. Get the principles first and then the details, if you care. As for reaching the top, there is none. There are too many directions you can take it.
But fixing it is a skill that usually comes with understanding. Knowing how it works in order to find the stages where it stops behaving predictably. Always remember that every detail was made by a human, to some degree. It helps with the assumptions you're making.