Yumezora's heart is so closed, Gomi's words can't reach her. Not he'd be able to have the right words to say, he doesn't really connect to her in any meaningful way, all they have together is sex. This situation is the first time Gomi gets a glimpse of Yumezora, but he got swept in her pace; "Let's break up" => "Love means nothing" => "Can you really prove that you love me, when you were happy to accept friends with benefits?". Gomi just panicked and responded in reaction, he didn't really look at Yumezora, how weird her belief is, to notice maybe, she didn't have a really good past to think the way she does (well he does know her husband is violent but he doesn't seem to process it yet).
And then we have the husband directly accusing Gomi of having an affair. I don't think he'll handle the situation well, his life will tumble down for sure. He can't lie his way out when his hostile coworker will rat him out for sure. I wonder how Gomi will proceed? Since it implicate both Yumezora and him, he may try to protect her by accusing the husband of being bad to her but it'll just backfire I assume, the husband will be convinced Yumezora did cheat with this guy who came to know their private situation, and for Gomi's workplace, being outed as homewrecker is probably not viewed favorably.