Maybe it's just me, but I feel bad for the protagonist.
Imagine being told at the age of 16 that you were swapped at birth and must marry somebody you have never met before, despite already being interested in somebody else. Said fiancée turns out to be a spoiled airhead with little common sense who then proceeds to monopolize everybody's attention, including the person you're interested in. You yourself are constantly ignored despite being an academic top-achiever, dragged into zany schemes that aren't supposed to be your responsibility and occasionally hit by the family that raised you despite being very nice, well-adjusted and mature yourself. Your biological parents also don't really try to connect properly with you. And you're never really asked by anybody how you feel about all of this.
Maybe the author wants to tell us that Nagi is just your standard stuck-up stuffy jerk who needs to relax and connect better with the people around him, but the fact that he isn't a jerk at all kind of defeats that. In fact, I am of the opinion that he is so far the only character in the story not behaving like a jerk or an idiot.
Just my two cents.