I know this Story partly, made me depressed for a long time.
In the novel, the gramps actually explains his story. He was summoned when the present princess mother was the queen and summoner. But the queen's mom and pops were dead.
The gramps and queen fell in love and the gramps fought to monsters or/and rejuvenated the economy(can't remember how this was). The nobles didn't like a commoner hero getting power so they helped the uncle who already killed 5 of his wife. The uncle in secret raped the queen and forced her to sleep with him by backmailing her with the hero's safety. Finally, the hero found out after uncle became king, brought an army to fight him when in the throne room queen states she is pregnant and not kill the uncle. Hero loses hope and is framed as a betrayer and put in prison.
Stopped reading after this and had forgotten about this until now.