Given the way it started, I'm predicting it's a case of the "friends" taking advantage of Amane's naivete to use him more or less as a piggy bank/hangout spot; I also wouldn't be surprised if they also got possessive of Amane and tried to control who he knew and was friends with.
Sadly speaking from experience on that latter half. Didn't even realize until I was in my 30s that it was straight up emotional abuse.
I probably would lean towards some backstory like this. Looked like Amane grew up in a small community, which is often close-knit, but one can still find bad apples. And I think the "friends," with one in particular, probably got jealous Amane had a good childhood, and if this person didn't have a good childhood, it's like "If I can't have this, he can't have this either," basically making sure he keeps Amane below him.
I feel that's how the interaction goes at the back end of V5,
but Mahiru stonewalls him, which gives Amane the confidence to write the guy out of his life.