Think that whole "summoned from another world" might've been referring to the "previous realm" that has been mentioned in this story many times before and it was just explained early-on very badly. I mean.. why else would this series not have an Isekai tag at this point?
Otherwise, if he really is an Isekai guy it still wouldn't be a plot hole (could've been reincarnated as Lucifer III as a. baby).. but pretty awful writing choice to be ignoring it for so long since the beginning of the series.