Damn that's messed up. His mother was raped, ending up having a child, then divorced, and left her kids.
But i feel like this now complicates their relationship unless the manga adds more. So, guardian or whoever parent, were their memories altered too? Was the mom's memory altered too? The anime never touched on their famiial relations. Now that they added those stuff, there's no way they could have gone by for years without anyone question the brother's disappearance unless he also got their's altered. It seemed like they were a loving family. Is this part really just going to end like that?
On a serious note, rape really ruins people's lives so goddamn I wish the law can be further enforced.