If you've read Playboy magazines articles, you'd know that plenty of men have had issues with even threesomes, because the pressure is up and they can't perform well enough with multiple women. Wives arranging for surprise threesomes have experienced their men getting frustrated and mad because they can't get it up with more women.
And you can't know if you can handle it ahead of time, because you believe you can handle it until you're actually in the situation. Playboy recommends starting softly and gradually build up to sex once everyone is comfortable with each other.
A harem is a good way to become incontinent early in life.