Ah yes, slavery and brutality can be forgiven if you just have a party! Who knew it was that easy?
I think is a bit more nuanced than that.
I don't know how it is the better way to phrase it... Kakyu is smart enough to have screened not just the truly loyal to the past leader, but the sickos, too.
It is pretty clear that the last leader, with the probable exception of a few, commanded using fear and threats, even to those who were "profiting" from his actions.
Remember... he was a psycho, he killed his own father and most of his brothers when he was a teen, his last brother he didn't kill just because he was young enough that he thought he could just groom him in obedience.
It excuses what was done? Not even close, but... they are in a new situation, there is a sincere effort to change... and a first step needs to be done.
It is less about forgiving, and more about... letting it go, so that it doesn't continuously poison the possibilities for the future, in two or three generations, this all may be a thing of the past, and those that come can enjoy life to the fullest.