Well, they were specifically told that four people had messed with Confinement King's dear Princess Mai Fujiwara.I'm not sure if they've even bothered trying to figure anything out. If I remember correctly, they were only told that "someone" did a bad thing to the Confinement King, but just that alone is so vague it doesn't really help find the perpetrator.
Furthermore, they've suffered from the shock of being captured, starved, treated as cattle, and trapped in a dungeon with overseers who can kill or severely harm them at a whim. It feels like right now they've completely forgotten about the initial goal and are in survival mode—either figuring out how to become a warden and receive better treatment or trying to rise the ranks in their new social ladder and become a loyal cattle safe from being whipped.
Honestly, if they all teamed up and immediately started brainstorming they might have come up with at least an idea or two of who they were even supposed to find but since their overseers purposely broke them up by creating "wardens" and giving out indiscriminate punishments they were kind of screwed over from the start, resulting in all of them trying to destroy/break each other. Lol, the Confinement King's harem is slowly becoming fully automated.
So, it isn't really that hard to figure out, if you go by process of elimination on who had contacts with her Mai.