I suppose if either of the girls handled that situation with force it'd just cause unnecessary trouble. Pushing Soujirou without actually harming him was rude, but perhaps not at a level that warranted anything particularly severe like drawing weapons on him or something.
He did also leave when they declined his offer instead of just being persistent, so he made a surprisingly good call for this kind of character in a manga. Left before he actually caused the other side to lose patience and squash him like a bug.