I know very little about the topic, but comments on other manga have said that the family courts in Japan operate differently than in Western nations, and that shared custody is not a thing. One parent gets complete custody, while the other is effectively cut out of the child's life, barring any financial responsibilities and the like. If the mother won custody and then wanted nothing to do with the father, it's completely reasonable that he was unable to see her.