He wasn't upset that his men were being bandits. He was put off that they were being bandits to someone who came to retaliate.
On a side note, you lose the moral high ground when you steal, rape, and kill innocent people. You're no better than the people who put you in that position. Even worse...