Normally I always think it's an extremely poor choice to reveal an isekai transfer to the locals (unless it's a world where it's publicly known to happen and it's not necessarily a huge deal), but in this case maybe not so much. Simply because the MC doesn't care. He came up with the lousy, half-assed amnesia excuse, which alone tells just how much thought he's willing to put into it. None at all. Even revealing the truth to Elma was more about if she starts to consider him crazy than about wanting to keep such a strange thing hidden.
Again, unless it's even somewhat commonplace, it's difficult to see any pros in revealing it. In this case it achieved nothing, except for satisfying Elma's curiosity. That being said, he more or less sealed his path when he didn't care enough to come up with anything better than amnesia. If he had used something better, like what I wrote about earlier if not still something more plausible, he wouldn't even need to lie about it anymore and Elma wouldn't have asked about it.