Its not that people are actually holding him responsible in their hearts and crying for justice, the teleportation disaster wasn't impactful enough to actually get any of the nobles worried, its just a bunch of peasants that died after all. However, such a huge event was ample ammunition to drag him down, he had two choices in front of him. First was to deny responsibility and fight it out, but with all his rivals in nobility bearing down on him, he was quickly going to dry up the Greyrat family resources and goodwill. Basically, his son, daughter-in-law, and granddaughter would all end up penniless and with no status. So, he chose #2, accept responsibility for the disaster and let the punishment be limited to himself.
For those thinking "It wasn't caused by him, why is there any burden on him?", the answer is that nobody knows why it happened, which is great for people who want to harm the Greyrat reputation, they can just spread the rumor "The Greyrats were experimenting with new magic" and without the actual reason for the event coming to light, its very difficult to defend against.
Although that guy Darius said "Not Enough", he wasn't truly referring to how well the situation was taken care of, rather gramps' ability to shield himself from the actions others would take. That gloating face was of a predator knowing his prey was absolutely doomed.
This kind of political maneuvering is actually really common in real life, you take a small thing and blow it up until it seems like your rival is a terrible criminal that can never be forgiven. It doesn't matter what the facts are, it matters how much sway their words carry.