I think some stories the dad yelling was what ultimately allowed the spirit to take over the body as apparently when you get surprised, your spirit becomes weaker or leaves temporarily which is the opening which ghost can use to infiltrate your body
Wonder if Yayoi can use the spirit extracting method the whacky uncle taught her to save Eiko a dozen chapters ago to extract the replaced and place them into dolls