You know, I've always thought of the implications of these kinds of isekai where if you return, you return to the exact place and time you were in.
Does that mean the entire world pauses the moment you get isekai'd? Is it just some kind of hallucination that your brain makes up in desperation, hence the "you will die when you get back" to scare your conciousness away from "waking up"? Does going back create an entirely new universe just to recreate that same exact moment?
It's more an entirely disparate world whereby the two worlds involved aren't spatially/temporally connected to each other at all. So time flowing in one world doesn't affect the flow of time in the other world and vice versa. Of course, from the perspective of the world traveler time will seem paused in the other world, but actually it's that the two worlds simply aren't temporally connected.
For the instance here, when Saitou was isekai'd from World A to World B, it's likely that the magic simply created a space-time bookmark in World A such that Saitou will return to that bookmark if he ever is returned to World A. Thereafter, regardless of how much time passes in World B, the only factor about Saitou that affects World A is whether he ever returns; if he returns, then events will proceed like he never was isekai'd, otherwise he simply disappears from World A entirely.
Though this also begs the question of why the magic isn't able to specify different space-time coordinates for Saitou to return to in World A, i.e. a different location or time period. But that's probably too far removed from what the mangaka cares to achieve for the scope of this manga.