I have a feeling we're going to find out that the story he's been fed about his parents is nowhere near the actual truth. We'll find out that his mom wasn't sleeping around and being a whore in either the literal or metaphorical sense. Instead it's going to be that she fell in love with someone who her father would've never approved of. She then made up the story about it being some random guy's child to prevent her father from using his money and connections to destroy the father for daring to sully her family name and then ran off with him first chance she got, leaving her kid behind either because her father forced her to as a condition of him not hunting them down or because she believed that a life spend hiding from her family's wrath would be no life for a child. And the grandfather then runs with this story and twists it further because it's another way he can manipulate the grandson (as evidenced by his attitude of "if you want to prove yourself as worth more than your whore mother then do what I say")
For bonus points her love could turn out to be an idol or a mangaka or something, creating more incentive to hide his identity from her family (since the scandal would ruin him more than if he was joe average salaryman) and giving a through-line for his otaku hobbies because his mother having the same interests is how she met his father. That seems less likely though and unnecessary beyond the more broad outline of my first guess.