Him hesitating isn't that important as compared to actually standing up to the offender. That's actually pretty commendable, a standout in the current herbivore-filled Japan, where making a scene is also looked down upon (especially with how timid he is). Basically, the victim's enemy isn't just the offender, but the surrounding people. Not sure if the critics here are in their teens or just narrow-minded westerners. But I get where they're coming from, anyway...