Judging by the fact the guy's armor didn't crumple under the impact of the chandelier, there's actually a somewhat reasonable explanation for what happened if you accept that Karate Baka here punched a Minotaur to death.
I mean, sure, the author probably didn't actually think of this and it's actually isekai magic, but let me have my fridge brilliance moment.
Standard Steel typically used in medieval armor buckles or crumples under force. This is why cloth gambeson was worn underneath it, as a form of trauma padding. However, Although rare, some plate armors in the later medieval periods were made of sufficiently Hardened Steel often wouldn't crumple, instead it'd absorb a lot more of the kinetic energy and shatter. Judging by the lack of Mr. Knightface's armor being caved in at the back, we could assume his armor is this Hardened Steel, in which case there was a bit of a genius idiot moment in dropping the chandelier on him and then goading him into a head-on clash: the chandelier caused severe micro-fracturing in the armor, all it took to break it entirely was a sufficient jarring.
Now it looks to everyone else that he can punch through heavy armor, when it'd really just be Physics at work within acceptable 'suspension of disbelief' levels if you accepted the baka (I don't remember his name) was super-humanly strong in the first place.
Now if only the author would be that creative.
(Of course, this doesn't explain his 'iron flesh of sword resistance +10')