People aren't wrong in stating the power of words, but let's be honest here; his fiance isn't wrong either. Those words clearly impacted him and made him change, but his dismissive "that's not your concern anymore" was just brimming with barely-concealed hostility. His father might be a sore point for him, but that reaction suggests that he hasn't made as complete a change as we might have been led to believe. More than that, there is the simple fact that he has always had a choice.
He claims he "doesn't deserve to love" after what happened, but all that really does is put the blame on his fiance. Like she said, he's the one who has refused to well and truly face himself after... Almost twenty years now? Trauma is trauma, yes. But he's still a grown adult who has repeatedly chosen to run away from any prospect of romance ever since. By all indications he's never even tried after what happened with his fiance. And that's ultimately on him. Because however painful a trauma, what we do with it is what defines us. What he's chosen to do is essentially to wallow in it, using his mistakes as a justification to never move on and try again.
And in her own bitter way, I feel like the fiance is kind of trying to help. Her internal monologue shows that deep down she was hurting from the emotional neglect and unable to face those feelings properly, so she lashed out at him in the process. Calling him out on letting those words define him for all those years since is probably the closest thing to a kindness she can afford him. Because remember: the characters do not see what we see. She hasn't seen just how much he's changed since then or how he treats the people around him. She only met him again for not even thirty minutes, during which he was just as frigid towards her as she was towards him. So to her, even with the FMC's account, he's still that uncaring callous jerk who neglected her and then just walked away.
It's my take that the author isn't understating the impact of words, but is also establishing that this guy has also been wrong to allow those words to shackle him in place for so long. That in using his "sins" as a reason to never allow someone to get close, he's in fact once more hurting someone close to him. Which multiple other people have called him out for. At the end of the day, as his ex-wife said, he needs to genuinely face himself and what he did, and come to terms with it.