Realistically, I can't see this turning into a "Sei was the bad guy all along" thing.
He's clearly gone in the head, and can't be considered an accurate narrator for his own story anymore.
I feel like this dude's going full inception mode with his delusions.
What's Yotsuya's deal?
In the beginning he cared more about clearing the quest than helping people of that world, as shown when he was willing to abandon the Arterosians until he realised they may be the "cargo", but now he's going the extra mile to cut down on the mafia in Goldia, in the name...