Kiryuu Shino (one of the central characters in “Kanojo mo Kanojo”) once said that there are numerous demerits of polygamy in Japan, not just being legally prohibited.
That would be an extremely roundabout way. They should have, at the very least, amended their Civil Code (article 732)...