Calling it: the dude that can do magic and is pulling the strings behind the narcissist daughter of the duke, is actually the child of the first concubine and the mage. Or maybe he IS the mage, and has retained his youth. Either way, he now plans revenge of taking down the empire's monarchy because they kept his lover (or mother, if it was the other scenario with him as the son) captive.
If it's the son, it may be more along the lines that he was born in secret, and is vindictive that though he was born of royal blood he is not in line for the throne. So by keeping the prince from marriage and relationships, it brings criticism on Caramel as the 'crown prince' because he's not fulfilling his duties. I believe they mentioned that particular criticism from the nobles in the beginning, part of the reason the king pushed for the engagement. In a plan to bring down the prince and appeal to those that oppose the royal family, the mage is trying to de-credit him? Or maybe to install his puppet, the duke's daughter, closer to the prince to then kill him.
In any case, I think the concubine's lover has some relation to that mage antagonist, what with the similar colored hair. And magic.