She's actually living in the lord's mansion, not the Dixon house.
The owner of the brothel, and with those words, clearly she met him back when he was active in the capital.
It was hinted earlier on that Noir is probably an isekai character himself, not really a reason so much as maybe a hint at what Noir may do later in the story. From what I've seen throughout this story is that the author may be a bit lacking in the display of relationship advancement, but where...