Well, This kind of pisses me off.
The story is weird, when thinking about feelings and this simple question,Why didn't Yoshida kicked his ass out? Well, sadly for both Yoshida and Sayu, they're both doing something illegal and that scum was clearly stupid and enough of an asshole to call the police, even if that would brought him down too. In a similar way, Yoshida couldn't call the police because with them, there would be a lot of questions such as "Are you his parent? - Why are you keeping her?". Without a trace of doubt the police woult take custody of Sayu and take her back to her family, that was the risk about taking a violent measure about all this situation, sadly for us Yoshida had to speak instead of taking actions.
This is a mere conjeture based on all the previous chapters and how the author draw this chapter. If we pay enough attention, Yoshida made a great effort by containing himself. Several times he clnched his punches and they started to "tremble" all over but he held back. Now, with this two things in mind (that yoshida held back due to police issues in the future) the real problem for all this drama lies in the lack of explanation fro the author. Everything would have been much more smooth and clear if he added just half a page explaining that issue, it could have been something like yoshida thinking for himself "We cna't get the police involved because they may take Sayu back to his parents", WE all know that Sayu doesn't want to go back to his house because of reasons.
Sadly, for us, Sayu is a shady character with a shady background and even if the asshole said a bunch of stupids things, there were several right on the mark, such as "Who the fuck she really is?". She lied to him before, why wouldn't she be lying to Yoshida too now?
The story is a mess, and as such, everything is and will be a mess as long as Sayu is involved