These guys are pretty much a bunch of baboons with all the swinging/casual sex going on. That aspect and love being treated very carnal makes any aspects of romance mean very little and also when the story tries to be serious like this chapter and the previous chapter. But I guess the author got to include these things otherwise it will be just a hedonistic hentai story.
This analysis is totally overboard for a erotic harem, but hey I can't help myself.