It took quite long for the empire to come and check what's going on. After all, the last group to go to the elf village to collect women never came back. It's possible that group was from elsewhere, whereas the normal tax collectors are from the fortress nearby. Since the fortress was raided by someone looking like a human, the commander over there had other things to worry about, aside from the elves.