I guess that the plot with MC's wife being distanced from her husband could be that she is also getting summoned as a Summon Beast to another world. And it won't turn out to be that she is cheating on him (or maybe she is cheating on him, but with a handsome knight or prince in a different world).
whil that sounds like an awesome plot point.
he would never notice it unless they meet as time seems to stop whenever they are summoned.
so i dont think that is the reason why.
My guess his wife was also a "summoned beast" either something happen, or she just want to keep that secret thus why his wife avoiding him, because feeling guilty, my bet she was summoned to same world and someday their will crossed path.