Are you serious, he was ostracized because of his voice? These manga authors keep making dumber and dumber excuses to let their character have a "sad" backstory.
@XenonXYZ I can see where you're coming from as I also believe that male Komi-sans reason for loneliness are kinda bullshit, but when reading fluff Manga like this you shouldnt expect above average backstory
"My body kept growing taller and taller"
*next panel, staring at hands*
"But my hands, my hands remained tiny, they were like an infants, and the other kids. Well, they laughed"