Yep...go...rescue them...for as quiet as you were originally trying to keep it, if that spy's accomplices heard the commotion and eavesdropped while you were arguing, they already know to expect you, and the hostages could already be dead.
But...plot armor will ensure there are no accomplices in the area that actually evaded Set's search.
This sounds like a rather resource poor operation, considering the empire is trying to blackmail a random (enemy) adventurer into murdering a specific target. If they had more resources, they would use their own people. In the first place, the whole thing would only work with a useful idiot because why on Earth would the empire ever allow the blackmailed, forced assassin to be reunited with their family? A decently intelligent blackmailed person would recognise that truth.
So, no, it's actually somewhat plausible there isn't another observer observing the observer. Even if there was, it's unknown if the second observer had ultra fast means of communication. The whole plan depends on the hostages still being alive, against the odds, so they might as well go forward assuming it really is an uncomplicated operation by the empire second-rate spies. Otherwise Elk should start looking for a gravestone.