I never said they(except the black mage) said anything bad about him. It's their lack of defense and objection, especially to someone they're indebted to and respect. They feel bad about it yeah, but still zero objection to something they knew was unfair.
You can't say that they're afraid of arguing against a decision when they've done it before.
And the fact that the black mage managed to convince them to switch her rarity with Kikuru means the opposite could be done with the other two.
But alas the running gag must go on despite how bad it makes these characters look.