The wording in the grandfather's summary of what happened with the attackers is vague. Did only the attackers get found of human trafficking and arrested? Or was the entire lot taken in?
One can assume that because it was mentioned they indeed worked for the villainous employer, he was also taken in. Considering also he mentioned earlier he could and would enslave ojou-sama.
But the attackers could be a completely separate sub-group, whom the villain employs, that trafficks.
I guess we'll find out another time.