I'm not sure why people are complaining, you all knew from the start that this manga is going to be a "MC is abandoned/tired of their team and leaves, and team is weaker without MC" type of mangas.
Even if the story does deviate a bit, the main chunk of the story is still his past and how he overcomes it.