Damn, must be a good chapter seeing 100+ comments...
Instead its because some people lack empathy, wow.
We the audience know she's the real deal but HE doesn't know that. His experience tells him to distrust. One of the girls, or perhaps multiple girls and it all added up... it must've messed with him really badly if he's still distrustful like this. Perhaps one of them did the long game like Izumi "seems" like she's doing.
If he lets his guard down again and Izumi betrays him, I know and he knows he'll NEVER recover again. He'll NEVER trust women again. Every time, every woman has been using him just to get closer to his brother. Why would things be different now?
His trauma is so bad that if he ever overhears Izumi talking with the girls who want to ask about his brother... his trust for her is going down the drain even if it's not her fault. Hinagi might be self-serving (for now), but she honestly saved them.