It's rather clear cut as to why he only has trust issues towards attractive women and no other group of people.
Because to him, it was only an an attractive female classmate that hurt him. You might feel that simply being called "fat and ugly" is nothing, but for people with very poor...