I figured out why this manga is so hard to understand. It's something Plinkett talked about. The characters already know the other character's backgrounds and motivations, so they don't say it out loud. What you need is a fish out of water character to have things explained to, so they can be explained to the audience.
The author had the opportunity to do this with Gremory in chapter 3, but instead he skips over it. So now there's no opportunity for things to be explained to the audience in a natural way. The characters are being introduced one by one in detail which is good but until then, the characters are bland and their motivations are unknown.
For example, why is the mask woman and Veracruz the ones who can't come to our world due to a lack of magic? Why are Nia and the Cleric the least affected? Because they don't use magic? The cleric refers to the king and queen as her parents so is she a princess? So then why does she address the vampire looking woman with -sama? Is the vampire looking woman a vampire? Why was the vampire woman a shut in? Because vampires can go out during the day?
All of this wouldn't be so unclear, and it'd be easier to be invested in the story, if we got a one-page biography dump on each of the harem members as a conversation to Gremory.