Wow this mc is self deprecating/modest to a fault. And this goes beyond the usual "no way she's interested in someone like me" zero confidence harem mc type. This applies to his livelihood as well.
Even though he played a primary role in defeating the demon lord, he pretends like he barely even helped. Why? One could argue he doesn't want to be rich and famous and just wants to live a modest life. However, his wish is to become a guild master of a guild which is antithetical to that. Guild masters are public figures who work with everyone including the city/government, the guild's members, and clients alike. If he wants to build his guild up, he has to make himself well known. And here he is pretending to be a drunkard when a client comes requesting a service. Guilds are essentially a business, and in business you don't want to make your clients feel like they're being disrespected or their requests made light of. How do you think people would feel if the boss of a company they're doing business with pretended to be a low level employee during a meeting? The princess went along with it because she knew who he was, but other people wouldn't look at it the same way after learning the truth.
Combine that with the women longing for him, despite knowing that's exactly how he's like and even liking that about him, and you got yourself a recipe for annoyance. Maybe there's a good explanation for this that has yet to be revealed, so I'll give it a few more chapters.