He don't want to be bothered by associating with the First Princess, but punching the Second Princess in the face inside of her own castle isn't a bother at all. I mean, punching a princess in the face in public obviously comes with no repercussions at all. MC logic is astounding.