Well, Portgas D Rouge was Gol D Roger's lover. Further more, she successfully hid Roger's unborn child from the World Government by carrying him for 20 months or so in her belly, outsmarting a huge faction of the Government.
Then Portgas D Ace made himself a largely known Pirate, and ended up joining the Whitebeard, which was considered the strongest pirate alive, and thus a huge threat to the World Government.
It is hinted that the D thing is an ancient thing, but maybe it's something that was born with the efforts of D carriers from Gol D Roger's time, such as Jaguar D Saul being a vice-admiral that turned on the World Government to aid Nice Olivia and then the "Demon Child" Nico Robin, Garp the hero of the Marines, a ridiculously strong marine with a strong sense of justice, yet highly unpredictable. All of those D's might have given them something to think about, and the Will of D was something that people would whisper from ear to ear, and then became a thing.
