Thanks for the chapter!
Did he loose his memories of Tachibana being a man before they came to this world? Because that would make sense, if you have a hole in your memory and your brain has something to fill that hole with, it is going to fill the hole in your memories. So it could be that he still has all of his memories, just that Tachibana was always a woman in them.