It's so insane they stay in the very town where they both lived and from where she was shipped off toward the capital. There's any number of people who would recognise her, if there's a moment when she's not under Hikaru's shared stealth effect. Including random knights patrolling the streets. There are also too many loose ends. Of course there is a little bit of truth to the claim that the kingdom wouldn't expect her to be hiding in the very town where she was imprisoned, when there were plenty of opportunities to transfer her anywhere else. However, the other truth is that the knights don't exactly have any leads pointing anywhere else. Hikaru, in his arrogance, didn't even bother to leave behind any fake hints. So, when the knights haven't got anything better to concentrate on, of course they would pay attention to Pondo, trying to see if the culprit left behind any traces.