Even if you do have a shield with big defense stats and your defense is thanks to it OP, even in knight times those who used sword and shield used shield to fight around and bash and mash and generally use it as both defense AND offense weapon.. swords get dull way too fast in combat compared to, lets say, axes.
Which is also one of the reasons why most vikings used axes by the way.
That said, nobody says you have to punch your enemy to death, i do think the hand where you have your OP shield will work much better, but then again, manga logic applies, so everybody takes it as given shield is only for defense.