Just because he is their master, and few of them did tempt him only a few times, doesn't mean he is physically interacting with them.
Not to forget, the MC official girls are actually against MC taking more than 9 wives, and they have been actively interfering with MC getting close to other women, including the Androids.
I had read the Light Novel until like Vol.21 (aka., the Honeymoon plot), and there is no such aspect in the story. Especially since the plot from Vol.22 onward involves MC's children coming from the future to the past to met the group.
They are just there as background support for MC, and practically nothing else.
Just for reference, the Manga raw (which is at Ch.108) is currently covering the near end plot of LN Vol.9.