We can view it in many ways, gray's relationship with silver.
A) he can be his father. As we know, it has been hinter/referred that tartaros are all demons and that they were once humans who were given a curse thus making them now demons. Now, gray's parents were supposed to have been killed by deliora, a demon that was slain/sealed by Ul. However, what if they didn't die and, because he was actually a strong wizard, Silver, grays dad, was actually captured and transformed into a demon? It would explain why he, as a demon, is a demon slayer himself and why he was described as loathing his demon form and preferring to stay in his human one. It would logical that this is the case, considering that a demon was responsible for his fate and might have killed his love, gray's mother. As for why he hates or dislikes gray, we can consider that it might be because gray's mom might have died to save gray who was too weak to defend himself. Silver would hate gray for being the responsible one along with the demon for his love's death, even if he is his son. Twisted love or something.
B) He can be his brother. Most of the above might be still true, except Silver might bee his brother. When their parents were slain, Silver might have been directly or indirectly resposible for it in some way and hated how his parents died to preotect gray or something of the sort. And that can be joined with hipothesis C)
C) He might be UL's lover. We cannot forget that Natsu might be smelling in Silver, his magic, not him. Gray's magic, UL's magic, Silver's magic...they all smell the same or similar as they are all similar. He might be simply Ul's lover and knowing ul died to save gray he might loathe gray because of it. Or, if he his gray's bother, he can still be UL's lover and hate gray for the same thing, blaming him for his parents and his lovers death. We can even consider he might be UL's brother, thus his hate for gray. Many many relationships can be explained.
D) none of the above. The most likely scenario is that this is a relationship we can't predict or explain. Brother, Father, Ul's Brother, etc Difficult to predict. Heck, as far as we might know, Ul might be Gray's mom or aunt or something even more complex. She took many secrets with her so who knows? But from gray's past, for silver to be hating gray, its most likely linked to Ul. Regardless of anything, the only real bad thing gray did was let ul die... if we view it in that way. Or Ultear, as she is alive but described as dead.