what is there to prevent a libertarian nation from degenerating into various lawless feudal kingdoms?
A population that embraces the notions of a Constitutionally limited Representative government that is allowed to arm itself.
If any State were to attempt to usurp its own Constitution amidst declaring itself independent of the Nation, there would be considerable amounts of armed rebellion. How such a thing could occur without immense popular support, I have no idea - but assuming the majority were simply complacent in the whole ordeal...
Ultimately, the Nation would be able to offer support to the citizenry to restore a Constitutional government.
Libertarian ideals hold that the individual is responsible for his/her life. It is our responsibility to ensure the government plays as big or as little of a role in our life as we feel it should (for Libertarians - that generally means the government be kept to as minor of a necessary evil as possible). It also means we are responsible for setting it straight when it oversteps its bounds or when some other force threatens the system or way of life that we currently live by and enjoy.
The proper question you should be asking is this: "What ensures the people of a nation will abide by its rules?"
There you will find the dangers of allowing small groups of people to make new rules, and the dangers of restricting the behavior and property of private citizens. History is littered with the skeletons of men and women crushed beneath the weight of an empire that would deny them everything but debt and service.