This character has never been shy about voicing what he thinks. If he really wanted to get rid of her he would and quite easily.After reading a second time i came tô realize she is a bad person abusing and using a good guy as a emotional battery.
There is nothing less or more tô it.
Case in point is literally this chapter where he gets pissed off and tells her off. There is no reason to believe he wouldnt just stand up and leave if he truly got tired of her.
You're looking for something that isnt there in this type of manga