People need to look at this from Valeries prespective
She wasn't welcome by the staff of the households because they believed Serenia is the true lover, only those who are close to the duke respect her greatly. They only warmed up to her just recently too.
The gossip about Kylus and Serenia was already too widely spread by the public, it is HARD not to believe it. Basically it's a public secret.
Serenia and Kylus calls each other with their first name. Something Valerie herself thought as somewhat personal.
Nobody in the dukedom never tried to make it clear to Valerie that Serenia isn't Kylus's girlfriend.
Kylus face is deadpan af. Valerie is just getting to know him. There's a limit in how much you can see someone throughly that personally in just a week or so.
I think it's impossible for the dukedom to not realize about the gossip between Serenia ans Kylus. So the fault lies within them too for missing the obvious social clues about a single man and a beautiful woman living together is shaddy af.