My current working theory is that this parasite does work on everyone. However, the catalyst for the actual growth is a person's desire to leave the town. Otherwise it doesn't grow but still affects their perception of the town.
Re-read a few chapters (specifically 10-13). It seems the...