I'm pretty sure the people at the table are students. After all, one of them is the guy who was insulting the other girl earlier.
But still, they are complete retards. Sure, go and pick a fight with the Duke, who's also the master of the magic tower. Why not? What could POSSIBLY go wrong with...