It's one of the points of his character.
He was not some bullied kid with no friends, and probably he had even a girlfriend. To him the whole "going to a fantasy world to get super powers and an harem of catgirls" is the delusional fantasy of a friendless lonely looser. He never wanted it.
But i...