He just needs to find his way back to a dungeon and he can tell them to take a flying fuck off the closest peer. Doesn't matter if he has to bribe one of his guards, scheduled a dungeon dive, or just do a bit of research on dungeon locations before blowing a few holes in walls. As long as he can reach a dungeon, he gets his full power back and there is not much they could do to stop him.