If they were from Yuzu's mother's side it would make sense but Yuzu and him are equally related to them
Nah dude, it did.
Just the fact that he is a mistress's child is really bad, though it's still a lot better than how people from Yuzu's mother's side think about him.
That’s why they keep his status a secret and only refer to him as a "relative".
There is stigma placed on how a child was conceived, though it depends a lot on the person how much they actually care about that.
For example, you face judgment for being born from a one night stand, an unmarried couple, a shotgun marriage, or a young mother who is still a student, etc.
I mean, look at how Koreans treat orphans or people with a single parent.
Even with legitimately married couples, if the mother dies in childbirth or one or both parents die while you are still young(it doesn't count if you are already grown up when they pass away).