Actually, if I remember the WN and LN correctly, Hero dude isn't even in love with Kaori (or Shizuku) he just kind of expected them both to just... Be there. Forever. Even after he married someone else, they were supposed (in his mind) to remain forever single and supporting him. It was honestly...