Honestly, the major issue right now is that when it's just MC and Chizuru, the whole story comes to a complete reversal to the beginning; he acts like he's never touched a girl before while she's basically indifferent to the entire thing. It's like for all of the development that we get with these two when other people get involved, they revert back to square one once they're alone together.
In other words: 1 step forward with supporting cast, 2 steps back w/o supporting cast.
Say what you will about Mami and Ruka, but at least they move the story forward and make it more interesting when they appear.