It still somewhat feels like justifying Grandfather's behavior, and I don't like that.
So you'd rather him just be a pretentious dick all the way through, with no explanation outside of just being a dick in general? If you're just going to listen to one side of the story without thinking about how and why things came to be, then you're not going to learn anything from their mistakes.
Like say there's a child who suffered through abuse. Said child grew up, had a kid, and started abusing their own kid basically as revenge for their own shitty childhood.
Does that justify their actions? Yes, since you got a reasonable explanation as to why they would end up that way. Is it a good excuse for them to do what they're doing? Hell no, no one would think that what they did was right at all. Just because we have something that justifies a person's behavior doesn't mean that you should be willing to excuse that person for being shitty.
Sympathetic villains aren't made for you to side with, they're supposed to be characters who had fell into pitfalls that you could take lessons from.