@Blazecam Time travel stories are really tricky to make easy to understand, especially when you just slide them into the narrative. "Back to the Future" adds a lot of establishing shots for the moves and the landings, as well as constant reminders about which era you are in.
Meanwhile this story only moves a couple of years through time, and tries to keep everything about the time travel (including who actually traveled - and how the entire mess started) secret, so I don't blame you (or anyone, including the mangaka) for being confused. XD
(Ok, maybe I blame the mangaka a little