After all the shit that you did to Rain, you guys expect him to go back and rejoin with the "hero" party? You didn't even apologize to him sincerely.
As expected from a bunch of people kissing the ass-cheeks and assholes of royalty and nobility, they were shocked that Rain would choose his life and happiness with his party/harem over fame and riches after defeating the demon lord.
Also, why the hell would Rain want to join the "hero" party when he already has a party/harem/family that actually cares about him?