Just like how Yuuki is being recruited right now, they may just search around for a good father candidate and share him together until they either say goodbye, escape or well, dies somewhere?
The Azumas have/had a very cutthroat attitude in many things, but I don't think they would kill the men or just let them die, even for them it sounds... disrespectful, I don't think they would do something so low.
But I am kinda convinced at this point that it is like you said... they find a good candidate, woo him a little, and pair him with whoever they think is best, but the men don't or aren't allowed to officially enter the family.
There is a fact that we should remember... just because we only saw the women doesn't mean there aren't any men that were born in the family, but Japanese traditional culture in this context has examples were members of a family that for some reason or another can't be in the "main" family simply form a "secondary branch" so to speak... second class members for simplicity sake.
And this... I completely believe they do, but surprisingly... I still think they would still be more or less well treated, as far as such a system allows.