It was mention before they have zero evidence whether Millia is actually evil or not, their only lead was from that adventurer in previous town, they don't want to take risk and get a bounty on their head so they trying to enter normally and investigate