Not sure whose fault it's (outside of story), but somehow I got impression of story trying to communicate me that Kaori was one of the bullies in that flashback and Yoshioka is doing this as a some form of revenge (although there might be genuine feelings be they lust or affection) an that...