In chapter 13, Souta's dad describes Elle as "a top secret matter", so whatever's going on with Elle, there's an effort being made to make sure it doesn't become general knowledge, but we don't know if it's the existence of "elves" that's the secret or just Elle's size in particular. We see in chapter 1 that Souta and Elle knew each other for at least a brief time as kids, and Elle had long ears even back then, so Souta himself has reason to consider that "normal" regardless of whether or not it is so in the world at large. Also in chapter 1, Elle laments having "grown too much", so it's possible that elf-eared people are normal in this world but Elle's size is still abnormal even for "her own kind" and she's truly one of a kind.
Elle's backstory is still generally a cipher, and honestly the fact that she was a childhood friend of Souta (though "friend" may be an overly generous term) who then left for presumably around 10 years before returning as a servant itself raises a lot of questions even without taking Elle's size into account. Were Elle's parents themselves servants to the Kaneari household? Is it normal for upper class kids to play with the servants' kids? If Elle's family was another rich family, then why has Elle been "demoted" to a maid? Does that have anything to do with why they disappeared for so long? The original story that this is based off of was too busy being a porno to explore any of those questions, and I'm kind of hoping if this goes on much longer that we'll get more of a look into Elle's family and past in general.
Update: After rereading the original, I noticed that Elle didn't have elf ears in the original, nor was there any mention of them knowing each other as kids - those details are exclusive to the "remake", which adds a further wrinkle to the intrigue.
