I'm not quite sure how he imagines that he'd end up hurting her if he decided to get a bit more physical with her. I mean, what, does he think he's gonna jump on her and assault her like a wild beast? Granted, it does seem like in the Japanese mind (or at least in manga) all men have raging uncontrollable urges that immediately leads to brutal rape if they don't maintain at least 20 feet from women at all times.