I'm guessing she got annoyed at the expensive gift because it feels more business-appropriate? Like a monetary gift that feels more like a transaction than a heartfelt gift. She has received gifts pertaining to her interest (tea), which are more personal, i suppose. Perhaps she sees this as a step backwards from personal gifts which show how well the duke knows her. Their current relationship is still rooted in the contract after all...