The big tells that say the teacher is a pedophile is the way he caressed Ririfin's hair on page 14. That was quite creepy and suspicious. And also the way Ririfin behaved, completely full of despair, broken even. i'd bet it takes more than just regular power abuse and being strict to make someone end up that way, even if Ririfin is weak willed. Saying it's because of the way the teacher looks is wrong though, it sounds more like preconception than anything else. Though actions, those say a lot about him.
Also, to those who didn't understand from reading, that's Riche's brother, she even calls him "Riri-nii".