It very likely has to do with their increasing "comprehension" of the Otherside and its rules and logic. Not to mention their bodies have literally adapted to the Otherside by now. It was explicitly stated that once you go to the Otherside once, it's more likely that you'll go again, but they've gone a lot of times, and are carrying pieces of it inside their bodies--wouldn't shock me that they're more likely to swap sides than the average Otherside traveler.
@LTalon they're also carrying around stuff they brought back from the otherside right now, like the rifle that the soldiers gave Sorawo parts for. Sure, the rifle was originally from the regular world but it spent a while over there.
The impression I get from this chapter is that the interaction with the cabbie was like the interaction with the waiter at the izakaya and they didn't notice. Nezokobama and Nadabaru are, as far as I can tell, not places in Japan - in the LNs the narrator makes a point of the fact that they didn't recognize the names. Nadabaru is a location in India so it's possible the implication there is that they were already going to be dragged across space to a deserted part of the world, and that's certainly where they seem to have ended up. At least there's no oil-crow (yet). We've seen discussion earlier in the series about human communication being mangled by spending time in the otherside, so it's probably the case here that both sides were speaking some degree of gibberish to each other and didn't notice.