Yea I mean I would question it too but he just so causally thought it that i was caught off guard, I thought there'd be more of a scene regarding it (also what about Hibiya? )
Iwakura will be embraced by Hibiya in the next few chapters. He's already shown he's a good guy and he treats her with respect. Plus it's a win-win since they both want to hang out with Kuroki and Chihara for life.