Nagi's real mother looks similar to Segawa. The best theory I can think of is that she was raised as a boy. And seeing how Nagi is in love with Segawa, its possible his father is trying to get his affection away from his real sister.
https://i.imgur.com/CtkujNV.jpg