Imagine trying to live a normal peaceful life with your girl and then she tells you that she wants you to have a harem, already has a set of rules ready, and also already has several girls lined up, including your neighbor and a celebrity.
Wasn't it revealed that their Earth is dying and that Justus is the only one who can save it? So he has to let the story progress to a degree where that happens?