@Aran1
Hahaha. I totally get what you mean!
Still though, I feel like playing devil's advocate.... I don't feel like rereading the earlier chapters to confirm this, but this chapter defiantly suggested there aren't any Servants there to great guests (Daisy opened the front door herself). In a stereotypical Noble society, Duke-san would probably expect to be greeted by the butler and some maids before he even left his carriage... not have to walk up to the door and ring the bell himself.
Confused/Concerned after a few minutes of just standing by the door with no answer, and given the impression he currently has of Daisy's Situation, its pretty reasonable to see him letting himself in and going to check on what is causing the shouting (his name getting yelled probably also would draw him).
And a Minor thing, I vaguely remember Duke-kun's people seizing the house and its assets in the original timeline.... with the father's debt to his casino, he may efficiently/actually/legally own the house.