Don't forget MC is a noble, and his servants are commoners, so if he wished he could kill them on the spot.
Thing is that was true even when they thought he was mentally challenged after the poisoning, everyone in a situaion like that would still treat him with respect openly and be very careful of saying something wrong with him around, if any they would be more careful than if it was a "sane" noble.
It didn't seem like villain grandfather knew about his descendants death yet, he simply accepted super fast that MC left them to die.
Not sure from the dialogue if he knew MC had lost no forces in the suicide bombing or thought that was just MC using the fortress at the end after the surprise teleportation when his forces were losing, the dialogue is not super clear to me.