Simple: use her accomplishments to undermine public confidence in him and his authority. Also, reveal to the public that he threatened to kill the man who has saved them twice who is also the Duke's guest. It's easy to do. Can even use Pero as a divine beast against him if necessary. At the very least, it will put him in a position where should he need to be violently resisted, he will be faced with the Duke and the public being against him and if he lies to his subordinates to get military strength helping him try to essentially kidnap his daughter, that would blow up in his face the moment they see her fighting against them.
Basically, grow a pair of balls and stop being limp-dicked doormats.
