Could be that, because Minato isn't a chakra powerhouse like Naruto/Jiraiya, he can't gather that much Natural Energy (although he's admittedly strong enough to withstand Natural Energy, so he's above average for sure). Perhaps the smaller amount of Natural Energy would explain why Minato can't maintain Sage Mode for long.
Naruto was able to gather Natural Energy more quickly in Bijuu Mode; perhaps having more chakra, and being able to gather more Natural Energy, also increases the speed at which one can enter Sage Mode.
Basically, I'm chalking up the inferior gathering-speed and duration of Minato's Sage Mode to his inferior quantities of chakra. Naruto has more base chakra than Minato/Jiraiya, and is at least as skilled as Minato in terms of balancing Natural Energy (and better than Jiraiya), so Naruto can enter Sage Mode nearly instantly and maintain it for long periods of time (at this point he's well past the 5 minute time limit, and can basically maintain it indefinitely). In Bijuu Mode, Naruto eclipses everyone except Hashirama; Hashirama is also a perfect Sage.