It doesn't have to be an exact time, but inspections would have come at a schedule, so the merchant would have to leave before then. There can only be so many routes to take in a city, especially when the merchant is supposedly trying to sneak out dirty assets. Even then, the bandits' lackeys...