They got mad because he refuse the offer of dragon worshipped like a god and spider considered as natural disaster but agree to some prostitute's offer. I think they thought of him as either a wimp or impotent and thus can understand but when he almost accept a prostitute they finally realize that's not the case. Cut the girls some slack guys.
Anyway I like that he's hiding the growing mana. The goddess might realize it and crush him before he grow even stronger. Maybe even using her "heroes" to fight him by making up some bullshit.
Love that the girls are hiding stuffs from him too because they are scared of his more murderous side. Like the one with Toa's case because she looks similiar to his old friend. They are scared of suddenly triggering him again. Which is why Tomoe didn't tell him about the head company's dark past of killing his own fiancee, her father, and her ex lover.
This is one of my most favorite isekai now. Dude is OP as fuck but e doesn't even care about anything and leave everything to his dragon and spider pet lol.. Death march and Kawazu MC make full use of his power to have fun vacation. Overlord MC use it because he's paranoid and wanna make sure no one can do anything to him and to find his friends.