Not to mention he wasn't really sneaking around. Yuzu was an active participant and the grandfather was there the entire time watching them and listening to their full conversation without showing any hint that it was wrong. Getting hit in that situation, I'd expect a teenager to run away from...