What gets me the most isn't that it's very specifically mentioned that they got married when they were both "not exactly young anymore". That's our explanation for why they never had children. No, what gets me is that it's also very specifically mentioned that Emelia pursued him for a long time until he finally gave in. Which means if anyone at all is to blame for them never managing to have children, it would be him. How long has he been carrying that regret? Of course nobody has mentioned it so far. No one who's been a regular long enough to remember Emelia would want to pour salt in that wound.
Selen is not just the missing heir to the diner that he was bemoaning in chapter 1 (where he tried to enlist the Demon Lord, of all people), she is the daughter he never got to have. Now we also know it wasn't for lack of trying.