I don’t understand the dislike for the protagonist at all, it’s perfectly natural to be uneasy and scared of letting your emotions run wild especially when you have literal Gods backing you up. In a sense he could also get away with freely murdering anyone he wanted to and claiming it was self defense because the dukes family loves him so much. He wants to separate himself from that dependence on their kindness to better himself, to be better than who he was originally. It’s honestly a good manga, I recommend most people re-read it with a clear mind and open opinions to really get a full experience.