@sketches
IIRC one of the "abilities" he got (that he didn't want) seems like a demerit to make have a bad leveling speed. The manga didn't mention it, but the novel did. I'm not so sure about the translation I read though.
However all the useless things are useful, and I think that's no exception. Notice how much his stats went up from LV1 to LV2? From what I've found translated, it doesn't go too far forward in the story,
so this is just my theory. What he thinks is a penalty to leveling is actually does make it slower to level up but the levels count for a lot more growth. He's killed a massive number of Goblins for example to reach his current low level. A lot of the Goblins out-level him considerably.
Now if the level difference between you and what you kill affects you getting experience, then he has a big advantage in the long term. He can keep growing from lower level mobs for longer than others. Higher level enemies may at a point be sparser, and of course they'll be tougher/more dangerous. And if anyone later on has the ability to see someone's level, they'll greatly underestimate him. Plus if there is some sort of level cap...
I'm uncertain if the mushrooms have any effect. Perhaps having more time between leveling up means time to eat more mushrooms, and the mushrooms you eat will add some stats at the next level up? That's only a guess. In any case he hasn't mentioned his stats in a while - only his level.