MC is suffering from some kind of bad social anxiety. It isn't so extreme it would have prevented him from working, but it has kept him lonely and isolated. I imagine he was bullied at school, which would have also contributed to his own sister, in the previous life, bullying him as well. He was too hopeful when getting transferred to the other world and possessing a local's body. He thought it would fix his psychological condition, but in reality it did nothing. He still can't interact normally with people, which means he couldn't join an adventuring party to effectively fight monsters. Forget about getting into romantic relationships. He can't even normally talk to other dudes, so women are utterly out of the question.
It's a rather sad way to deal with a mental illness, but slaves might help him to overcome some of it. His social anxiety is probably multifaceted, so standing higher in hierarchy (than slaves) could perhaps remove a sliver of the anxiety. But only perhaps and only a sliver, because he's still extremely nervous and keeps stuttering, as we saw.
