Yea, it is a batshit insane assumption. It is quite established that, neither of them, are, in fact, sane. Else the gay boy would've done what you said lmao.
There is also his house (made specifically to cater to his lord's taste) and riches, as well his connections, and as he seems to live...
It was an assumption based on their perfect memory of their past life (minus the Demon Lord battle) and impression of one's death would leave on oneself.
There is also the memory wiping probability, but since they cannot remember anything about the Demon Lord battle, this assumption won't come...