I'd have suspected the duke from the get-go, presenting himself as devoted to the Empress - why wouldn't a guy like that harbor animus toward the bastards of a mistress? That he's blood-related should have made them MORE suspicious.
Also, the second prince's guardian not recognizing the rest of the family has treated the twins like trash is annoying - I get his loyalty is to family, but he should have the wherewithal to say to his master, "Hey, Second Prince, the twins still think y'all think they're trash, and by sending them away you're confirming it. And oh by the way, because of this, they'll happily summon Ragnarok on your asses."