It must be a memory ≠ knowledge kind of thing. He knows information in it's rawest state, but how he got it, or things that happened because of it are gone.
So both future swordmaiden, and future observer decided to influence the other in the past. He did mention something about waking her early. I wonder what changes this will lead too.