I just don't buy it that this protagonist would care this much, just cause that one asshole got a little bit beaten up. It wasn't even all that serious, and like the guy himself said, it really wasn't in anyway our mc's fault. (It was all his own fault anyways)
That's why this whole duel situation feels very forced and needless (and silly).