@f0xhunt bit late, but the butler offers to call her back to the house, presuming that she'd come back right away
I guess the Duke was being perceptive enough to pick up on some disdain for her from just that, i guess his tone was probably bad too
But yea, the most a servant should be offering is to notify her that the Duke is waiting, so she could decide whether it's worth coming back for, but I'd imagine that they'd usually wait for their masters to come back before informing them of what happened while they were away