Seems like some people don't understand who these protagonists are. These guys wanted to die and have found a will to live. It isn't something that is accomplished so easily. Even if this guy clearly is threatening them. It wouldn't be a smart move to move in so soon when ill prepared.
How are these people so passive? The antagonist is clearly threatening them and is alone. They know that he is feared by his followers, so if his life was to, say, expire, no one would seek vengeance against them.