You're looking too much in to it. We're talking manga physics here.
Where rubber is stretchy and diamond is hard. If you want to take this so serious maybe we should ask ourselves why Luffy has never snapped like a rubber band. Because that is the fate of rubber.
If Oda depict Jozu as a diamond man, he is saying this is a tough guy to hurt, this is one of the best defenses in the world. Brook can heal his bone injuries pretty fast by drinking a lot of milk. That is another hyperbole of a real world appliance used in the One Piece world.
We can agree to disagree but I don't think that Oda would make Jozu brittle to blunt force.