Parish houses are a thing, or we're. Might have been part of the early cultural exposure.
Also, turnabout is fair play I guess, do monks/priests live in temples? Cause there's a not small number of manga/anime where people who run shrines and temples also live on the property. Might just be...