The thing about Haruji is that he is as irredeemable as they come and a lot of the character merits people assign to him are what they would expect of a normal, well-adjusted person but not applicable for him specifically. It makes for a very one dimensional character and while that is a bad thing from a literary perspective, it still doesn't make him a good, or even a relatable, person.
This chapter doesn't spring anything new about his character, but only takes it the next, and predictable if you're familiar with his psychosis, level. None of the previous panels featuring Haruji and Taiga do you see him even regarding his son with anything approaching warmth. I don't think he even looks at Taiga. And when he's gaslighting Mio, it becomes clear. Taiga is like a puppy Mio decided to bring home entirely on her volition. The speech he gave when Mio told him that she was pregnant was not a permission to have a child together. It was an abdication of his responsibility as a parent. He's telling her that she will be the one deciding the have a child and she's going to be the one solely responsible for him, for good or for bad. They probably had similar conversation all through Taiga's life, because obviously, it's difficult to entirely remove himself from Taiga's life, and every time Taiga happened to be a hinderance to how he wants to live, he would remind Mio that having him was her decision, and she should be the one responsible for him and not him. Otherwise, his line about how this is all her fault wouldn't work, as it didn't work with Doubara. So it has been a lifetime of conditioning Mio about her responsibility toward Taiga that made that line in this chapter work against her.
I can't find any evidence where he's expressing grief about Taiga's death. He's expressed annoyance at how people are treating him, expecting certain emotions that he's not capable of feeling. The admiration and respect he expects from other people, along with the lack of empathy, pretty much clinically diagnose him as a classic narcissist. The manga fills him in with some details. The encounter with his family shows that his family is one of some former renown, maybe a landholding family that has fallen on hard times, surrounded by families of former retainers who still pay them some measure of respect but are doing better than them. It's a common theme in Japanese drama that native readers would have picked up on immediately, but something not that familiar to Western readers. That decline of his family mirrors his employment situation. From his uniform, he works in some manufacturing job, which is perfectly respectable, but not to the level of an idle landowner that his previous generation was. Or not a professional occupation like a lawyer or a doctor. He's not even an owner or a manager. He could not be happy with his family or work situation, given the delusions of grandeur typical of a narcissists. He was probably satisfied when he was in school, but that has steadily declined after graduation, a pattern that should be familiar with the Western readers as well. He's like a former high school jock who still wears his varsity jacket to bars at 36 and hits on the waitress 15 years his junior.
So he can't handle the pity from people who he feels should be admiring him, and now one thing he feels he has that's (not who's, because I think he sees Mio as a possession) better than what others have, a beautiful and popular wife, has run away from him. All this he sees an an attack on his person and he can't take it.
People are assigning grief to Haruji based on how they might feel in that circumstance, but I don't think Haruji is most people. He did not and does not feel grief about Taiga's death. That I can say that with this much certainty because this manga is a made-up story and the ones involved in creating this manga specifically chose that narrative. Again, it makes for a bad and restrictive story, but it is how it is. Haruji is a bad, irredeemable person because he was written that way. I mean, no one said this was War and Peace.