Hmm I don't really like the mc attitude. He cares too much what other think because he hanging with a beautiful girl. Let them be jealous!
Remember that this is high school. This manga is pretty low-quality, but his reaction isn't
that unreasonable. You absolutely should care what others think to
some extent in all life situations because what they think
can result in consequences. This is a romcom manga, so nothing will happen beyond jealous nameless guys seething in the background, but in real life he could get bullied or ostracized for being friends with a pretty girl while not being among the popular kids of the class. This can happen in any Western country, so just imagine what it must be like in Japan where conformity and "knowing your place" are even more important. Obviously, he's taking it a bit too far with wanting to avoid her, but again, this is just a manga.