Exactly! This isn't "THAT BITCH OF AN EX WIFE NEEDS TO BE PUNISHED," it's establishing that he sees his size as intimidating, his strength as dangerous, his company as unwanted and his life a drag on those around him.I feel like the setup is less "women are bad!" And more to establish a setup where the male lead is incredibly traumatized and beaten down. It's a plausible way someone could absolutely destroy his life in a way that leaves him with some fairly severe PTSD. Note he blamed himself for the tub falling apart and immediately assumed someone who has shown nothing but positive intentions was going to instantly turn against him. Part of this story is probably going to be him learning to get over his trauma and learn to trust people again.
The scene where he remembers his wife covering her head and looking scared is the point, not that he did naht hit her, he did naht.
I think the manga has done a good job setting this up! He instantly accepts blame and even says that he doesn't remember hurting her. He doesn't viciously deny it; he doesn't think its true, but he also, clearly, can't argue that its false. This isn't gonna be that kind of manga, but I can even see a development where he did make his wife uncomfortable and threatened, but not out of an excess of malice. That would be a fascinating turn; if her feelings were legitimately felt, but were mis readings of his intent, it makes the whole situation more tragic because nobody did anything wrong. If there's a villain, its easy. Which is why I think the manga is mostly fine with making her be completely in the wrong, because it can then move on to the story about changing his self-perception.