Admittedly the Earl was being an ass but he was also doing his job, he was ordered to bring Jin to the Castle. That won't change no matter if Jin is the culprit or not. Jin was also willing to go, but the villagers just weren't having it. Technically they were commiting treason and rebelling. So in a way the were asking for a fight.