It's not like his childhood would have been especially horrifying, but it would depend on his personality how it would shape his view of human relationships. He just happens to have the kind of analytic and negative personality that would grow up this way under such circumstances, plus he doesn't have much drive to get close to women. Maybe if he had moved out to live on his own as soon as possible, things could have been different, with time healing wounds to a degree. But he's still living with his mother and thus his impression that relationships bring more burden than benefit is constantly reinforced.