He didn't realise it was an issue until they were already inside.
In other words, he doesn't think ahead
at all. For a guy supposedly in charge of chores in the past, how he even got anything done is beyond me.
Also, he had this same problem when fighting in the middle of the city, and he used that fire sword. Why is he even panicking now?
Overall, this manga is quickly devolving into bland progression with overuse of tropes.