"...even with this explanation, I still don't get it."
I don't think it is that complicated: He tried to alter history without altering the course of history. So he saved the elves, secured the magic stone and made it seem as if the village was destroyed, Esther's and Fury's memories were overwritten by the elven elder (on Sekai's order) to fit the storyline. Fury then did to go on a journey and meet as per usual Sekai. When the story moved to the elves village ruins, Sekai would use Shinra to travel back in time closing the time loop. Afterwards Fury and Esther got the big reveal "the elves are all alive and well" and how Sekai is imprisoned by the association.
Honestly not too complicated, but considering that this is a montly manga and this time travel/elven village stuff spans ~20 chapters, it is too much to comprehend without a reread, the illustration alone is lacking.