Seems like this is one of the manga where seemingly important character details only exist for the sake of jokes. When they would be a real weakness in a critical situation, they suddenly don't exist anymore. I bet once they are back in town, safe and sound, the dude will again be back to his Zoro level of orienteering.