Their so called Goddess is the one who sent him there and gave him the castle, that man is basically like an apostle of said "Goddess" but I suppose there's no way they would know that. Also they have an identical statue of her but maybe she didn't bother telling anyone about him. I guess she did take his soul by mistake, she's the average useless "Goddess" after all.