This first started as your stereotypical female rivalry but morphed into trying to force it out of him and that is exactly what is driving his brain to "I want to leave this village", and whatever everything else is going through it. At least the story has actually stayed on track to some extent. I'm guessing this needs at least another 10 to 15 chapters to get him to say anything significantly new, at least from the girls' POV.
I can't tell if the author is stalling to get significant ideas (seen it many times before) or if it's just high grade harem torture.