Well, I though author will give us the excuse that they were not retarded but being mind controlled (that should make more sense in a way) but I guess it will be the 'everyone is and idiot but myself' in the end, feeh...
Also, if his curse/skill is being a loner and that curse/skill can power up like others skills when he level up, should not this story reach a point where all other people (or sentient creatures) will avoid him? Why he keep meeting people when the whole premise of the story should make him more and more lonely?