Didn't he say he was leaving that party? What's the insult here, that he followed through with what he said he was going to do, and was actively doing by not meeting up with them any more?
Also, why are they suddenly so weak? Because they really were, to some extent, doing the MC a favour by having him tag along when he couldn't do anything beyond carrying some luggage - if even that, given his weakness from the curse. So why are they suddenly unable to complete quests, when the MC's absence shouldn't effect them in the slightest? This is just plot conveniences and following tropes despite it making no sense.