Maybe that was a noble family's vacation house. It's kind of strange the village, despite being so prosperous, doesn't seem to have any administration, beyond the village chief, who's like any random villager. If a nobleman had actually lived in that building, ruling over the village, why would the family have abandoned the village? Or why didn't a duke/king name a replacement if that particular family died out?