The real point of annoyance with him thinking like that isn't necessarily that he is doing so, considering lack of self confidence or self esteem can do some amazing things to turn things into shit in your head. That kind of shit is super real but happens with filling in the gaps; when you're unsure of someone or you can't read a situation or stuff is left vague because you and the other person don't talk about it. Meguro isn't like that tho, she has done literally nothing but be 100% honest and blunt with him so there's really no wiggle room to misread things. Like how people get annoyed when a character confesses to someone and the person doesn't understand; it's not natural or normal, it's obviously done for other reasons and doesn't make sense or fit with the situation. It's bad writing. So when she directly says shit and constantly is only honest with him, him not believing her is really really shitty.
Hopefully he'll grow and he's already showing signs from last chapter and this chapter where he is standing up for her. But if he's always going to be a wet blanket then the relationship will feel strained and...fake(?) like it's being written to placate a fanbase or continue a status quo instead of naturally changing and growing.