Personally I don't think it's far-fetched for him to deduct how it happened. It's not like there are many more variables.
He's almost never at that place, the people he suspects have been acting suspiciously from the start from his point of view, he knows they are savvy, he's also (rightly so)...