Well there are various theories regarding past Gomu Gomu no mi users.
Some say it's Rogers(you know because ppl see Roger in Luffy) or it originally belong to her wife(as she kept Ace in her belly something like that).
U see Shanks believed Gomu Gomu no mi as a rare treasure.
We saw Gomu Gomu...