We can see why MC stats are the lowest in the world, instead of coming up with a good excuse he let himself be brough around from people that think of him as nothing.
A perfectly good excuse I come up in 10 seconds was "I found a party of weak people like me, we hunt for the weakest monsters and share magic stones fairly, no danger, no bullism, I'm not joining dangerous hunts anymore".
And in case they try to ask who are the members I would keep silent to protect people in my state, if they tried to stalk me they would come up with nothing and waste time as I would "never move from my house", could even say how I talked with my new party and they saved some stones for me until the bullies stopped trying.
But really, the core of this story setting is shit, forcing people to hunt monsters, even the ones with the worst stats? just put them in charge of supplies outside the danger zones, would make things better for everyone.