The right to have an affair rightfully belongs to those who can nuke a town or two under their own power. Those who cannot can only spit, sputter and complain at those who can. Physical Might and Magic make the rules.
Apply that to our real world, and the result ia obvious; no one in our world can nuke a town or two with their own physical might or prowess of their magical abilities, so no one in our world have the right to cheat on their spouse.
The right to cheat is not universal, it change from world to world, reality to reality, each according to their own physical law.