Seems like this is one of the manga where seemingly important character details only exist for the sake of jokes. When they would be a real weakness in a critical situation, they suddenly don't exist anymore. I bet once they are back in town, safe and sound, the dude will again be back to his Zoro level of orienteering.
Come to think of it .. if MC can imagine anything and he just needs the materials to craft it.. why not craft a gun? 🤔 weird that it hasn't crossed his mind..
Bad sense of direction doesn't mean bad memory or inability to make a map. Bad sense of direction is more about spatial sense and mental mapping. It can help a lot to actually map it out physically.
However, that map is WAY more detailed than expected, to the point where I wish they had pulled some kind of 'good thing crafters can also craft maps' BS.