Well, we don't have any backstory on why the MC can't speak the language. I remember mtl'ing some syosetu yandere novel when a girl forcibly summoned by the love interest to another world, then have a curse placed on her so she couldn't understand the language except when the love interest talk. When she learned a language, it became obscure, when she learned to write, the paper burned, something like that. It could be nice if that situation applied here, rather than MC reluctant to learn language, he can't for some reason.