"Did I overdo something again?"
Did you ever not?
"Are we under a curse that stops us from leaving whenever we want to set out on a journey?"
Yes.
"Didn't you say, 'I don't want to stand out'"?
Sure did, and as usual, didn't follow up on it.
Village of delinquent Karens.
I'd all that a town. They have a moat and seems to have enough speciality crafts for a town. And several inns.
That location is in no way hidden, unless it's far away from where people usualy are. But it looks like it has two sides of the building towards an open square, or at least wide street.