MC has super hearing. Eventually upgrades his Super Hearing to the point where he learns magic just by hearing of the magic. He's in an isolated tower in the middle of fuck all nowhere. Likely he is listening to magic being discussed in faraway towns, maybe even wizard towers isolated from society. Basically he's listening in on at least a general country wide area to learn all sorts of magic. But he describes it himself as The World.
Somehow did not listen in on all sorts of taverns, inns, shops, adventurers having small talk and learned about the general history, standards, slangs, common sense and general know-how of the world.
OK.
While I do think the premise is stretching belief a bit, it could simply just be that because it wasn't information he cared about (learning magic) he just didn't retain it.
What I think is odd is that he heard nothing out of the house about his mother passing before his father came to him and told him which may have been months after she passed. And she made no play to see him one last time before she passed on.