Japanese authors never let logic of any kind get in the way of their trope, cliche comedy. Of course the dude simply marches in despite knowing full well there are now a dozen girls living in his mansion. He could call from the door to check if there's already someone in. But nah, that wouldn't fit brainless ecchi comedy. The same reason why manga always makes it look like nobody in Japan knows how to knock on a door.
It's all even funnier in a story like this where the MC wouldn't actually want to draw extra attention to himself and would, for some mysterious reason, like to avoid getting killed. Despite that, the author is incapable of coming up with anything better than this. Trash isekai gotta trash.