If they took his magic stones to bolster the defences he should just treat it as his contribution to the fight against the ogre squad and leave it at that.
I still don't understand why he needed people to spread bad stories about him. Can someone explain to me why?