It depends on where you come from. I'm from Norway, so almost from birth I can talk and/or understand three languages: Norwegian, Swedish and Danish. Norwegian has a lot of similar words in German and English, so those are fairly easy to learn(to me at least) as well, though I didn't pay much attention to German in school, so I don't know it that well.
The more difficult languages would probably be anything outside of those, but the most difficult ones probably has to be Chinese and Vietnamese for me, that I can think of. They have very difficult pronunciations. Japanese is a very straight forward language, so it's easy to speak, but hard to learn as well.