Why doesn't Tatsuya just stop hanging out with either gang and calls it a day? If he's so worried about getting targeted by either, then simply cut all ties with them, they'll surely understand, not like he owes them anything or pose any threat to either.