The bias of the townsfolk against Elpis and Margit ensure they won't interfere. The city guardsmen, at least some of them, know the real story but are simply too weak to interfere. Remember, Elpis can go toe to toe with an A rank adventurer, of which there are only 8 out of 100,000 adventurers...