I think this is this manhwa's quirk: the backstory is really lacking.
We finally know the name of MC's brother in chapter 5, along with the Duke's current whereabout (not at home). And that maybe the Duke ordered the servants to mistreat MC (why else that servant could act so brazen when confronted?).