Yeah, I think they were just talking about how their party is struggling now that they don't have Remme to debuff their enemies. Half of the hero's party didn't actually know he was the guy they fought against until Remme revealed it himself.
Remme's "problem" with the group was he and Phoenix actively hid the extent to which he was helping the party (ch 13), so yeah, the two of them owe the others an apology, even if they had good reason to do what they did.