Would any author dare to have the protagonist reject women by saying "I'm sorry, I have taken a vow of chastity and I am a eunuch"? To me, that seems like a quick and easy solution.
If I remember correctly, when they mention the ban on carrying weapons during celebrations, they say it is absolute, regardless of identity or affiliation.