Considering the world the MC is currently from, do you blame him for wanting nothing to do with relationships, or women in general?
Is he a herbivore for wanting to live on his island alone and not deal with the bullshit of ranks? I don't think so.
I think he has a slight hero complex because he sticks his nose in business that puts him the spotlight, like the duel.
Honestly I don't see why he didn't just stay an adventurer and live off that money and not go back, but if he did, there wouldn't be this entertaining story.