I used to hate the grandfather, but after seeing his history I can't quite hate him as much anymore.
And I think I see why he treated Kyouichi and everyone else so coldly. Even though they were family, every time he looks at them he would have to remember the future that he wasn't able to create with Fuyuho.