As others have said the frustrating thing isn't that he hasn't become fluent in the language or something. Of course that's going to be a long, difficult process. The frustrating thing is that he's not even making an effort to learn anything. Try to learn a few words and start from there. Generally speaking nouns are easiest. Point at things, say their names repeat.
You can still have a manga about people with communication difficulties while they are making an effort to communicate. But this guy isn't even trying. He is constantly frustrated by the fact that they don't speak the same language yet it's like the idea that he could attempt to learn it never even occurred to him...