Shirou does have pretty solid defences against random troublemakers, so he ought to be safe in that sense. He also has his own house, separated from the town the newcomers will build, making his position still more safe. So, his only concrete danger would be from a systemic source, that is, if the colonisers want to put him into a proper place for a man (I don't mean the brothel, but to tell him how to do things, when so far he has been his own master). It didn't look like it based on the first contact, but who knows. Nevertheless, it's a good thing he has the bird golem looking for alternate locations.