By no means did he have a harsh life but his issue was that no one cared who he was, only the weight of his family name. Compared to everyone else in his family though, of whom we are told are all "elites", he's just a normal guy. So they use him for his money and talk about him behind his back. He even mentions trying to hide his background but people eventually find out and harass him for his ineptitude or just disappointed in how ordinary he is. Getting the isekai treatment is wiping the slate clean but yeah, handing him the silver spoon of demigod and doing dailies/weeklies for OP skills defeats the purpose of him striking out on his own.