Yup that's how I took it as well. The party with Est in it didn't really interest him, he was just killing time because of Athena reminding him of his lover back then. Now that he's got a little harem formed, now it's for real.
I don't feel like this series is taking itself seriously at all. Author is just having fun with a harem of MILFs... in fact author just gave MC the vitality of a dragon just to he could bone his harem of MILFs more.