The guard, Ginger, is definitely caught in the middle in this.
He is loyal to his lord, 'cause he's that kind of guy, but he also feels his lord was going to cross the line with Aisha, so he aided her mother in smuggling her out.
In other words, part of his loyalty is protecting his lord from his own bad judgement where possible. He's loyal, but not blindly so; he does recognize his lord's faults.
Now, he's got instructions he's really not going to be happy fulfilling.
It'll be interesting seeing how things end up for him.
And, in regard to Pax...
Considering what all he could have been doing to Lilia and Aisha, while it was kinda perverted, it was actually a fairly clean perversion, as it were.
Yes, it actually was a torture for Lilia, as Lilia really works on presenting a stern, somewhat aloof exterior, and what Pax was having her do was so contrary to how she prefers to present herself to the world, but... she's a Maid, she knows people have Maid fetishes, this was something of an occupational hazard that she was aware might come her way sometime. It wasn't the hentai Maid fetish, at least.
This is not to say that Pax is a nice guy. Not at all. But it could have been far worse.