Rather than physically incapable of saying it, he just thinks it's a bad idea. Granny jokingly asked if he might be from another world, but that probably isn't something they actually have on record in this world as a confirmed phenomenon. If he says "Oh I remember my past life, it was on a completely different world," he risks sounding like he's actually crazy rather than just poorly adjusted to common sense.
@Serenata, @Glomoro
He despised his old life, not really wanting to link this one to that. Plus he fears the people who love him might change the way they look at him.