In the first chapter it's says that he rememberd his first life right before he decided to spend his money on prostitute, he only retained skills before that, still flimsy argument but that probably wat caused change in attitude not ridicule.
(Autor might of not though about that though.)
I love how there is all of this talk about great wall but it will all crumble to a single cannon that the MC got from the the ledy he had sex with, lady's surely giving bater drops than bosses it this story!