As people said before for starters DF users are quite rare outside the Grand Line. There's a high concentration of them in the Grand Line and that's where most of the story occurs. Also you need to take into consideration that DF users are often shown, but compared to the total OP population it's still extremely low.
At a certain point an army of a 100 000 men played a role in the story. If you say a 1000 men of those would have a DF, that would be 1% of the army, however it's that 1% that would get the most attention. That number 1000 was just a number though and in reality it was probably far lower. A 100 would likely be a more accurate number.