@Kaarme
3 reasons, I would say
[ol]Age. As you say, he's in elementary school, they're not really in the most hormone-rich time of their life, so the suspicion is lower, but in several chapters his classmates behave older and thus more perverted, thus being noticed, as i'll get to in 2nd point.
Never showing any outwards reaction towards him seeing what he sees. If he were to stare longer, smile, whistle or do anything that might raise an interest in what he's looking at to make that reaction, people would notice it, and then realize he saw it.
The event itself. Many of those events is something extremely simple, which you can't really scheme to achieve, but you have to get lucky to notice, others might be the result of him being nice in some way, which gains him that event as a sort of reward for being nice, but noone notices the particular way it could be considered. [/ol]
Basically, he's a very young & gentlemanly lucky pervert. You can't plot all of those events, and if you do, you would react differently from him, thus having a higher chance of being noticed.