Removing aging as a whole is clearly such a bad idea.
If aging is removed the no living being can grow up, including humans, animals and plants.
No more new generations of humans when humans can still die through various other means, and no more renewable food sources from plants and animals is a slow way for humanity to die out.
Looking at the Japanese for this, the word used seems to be 老い, which rather than aging in general (growth from any age to a greater one), would more accurately be translated as "old age" (according to my Jisho search at least). In that context, I think the decision makes more sense here (even if there are other reasons why you might not want to remove becoming old). This also makes the motive of the old former minister even clearer here, because if simply aging ceased to exist, he probably would still be old and would just continue to suffer without end. If old age is eliminated entirely, he might turn young again (if he doesn't just completely disappear).
On the topic of the bad consequences that removing old age would bring, though: considering that Public Safety presumes they can control the Chainsaw Devil and seeing as they do have no hesitation in using his powers, clearly they're prepared to use him to deal with any undesired consequences (whether or not such a balancing act can be successful is debatable, but humanity is certainly not lacking of people arrogant/proud enough to think they can do it). It's even pretty much stated they intend to erase multiple concepts to create a (perceived) better world.
In fact, erasing the "Old Age" Devil first might be the solution to another problem and show how they're actually clever. Considering the coming return of the Death Devil, Public Safety is surely preparing to try to defeat them and possibly use the Chainsaw Devil to erase Death. However, if they were to do so without removing old age as well, a world of pain would be sure to follow for everyone.
This is actually something that is often overlooked about immortality and when storytellers remember that immortality does not necessarily mean eternal youth, they can punish their characters severely by having them age forever. If Fujimoto/Public Safety is actually foreseeing this problem, I salute them.