The potential difficulty in relating color information does make it an interesting topic to explore in a story centered on trying to learn an unknown language, though.
Oh, and there's another great example of color confusion in more-or-less modern Japanese: Prior to WWII or so, green wasn't...