The king himself.
Someone thinks his fun is above the king's fun.
It's considerate of him to think about who has an easier time visiting whom.
King has a good opinion of the prince, except he prefers too much peace. Prince seems good to me.
That "nobles are above commoners" might be true in their society, but I don't think he'd lose respect by not following those lines too closely. Yes, some nobles who're already scheming might get worse, but he'd also be seen as a king who's connected to his country, which is generally seen as a good thing. Meaning commoners and lesser nobility will become more loyal. Though exact reactions depends on culture to a greater degree than we've seen so far.
Then he got walled.