The problem is that you now are letting others steer your opinion on the matter, abandoning your taste because of the opinions of others.
Also, I hate the term "The fandom", they are just a random group of people that get elevated into representing the community of a media as a monolith, but you just can find people who also like things the same way you do, these losers should not in fact determine your opinion.
It would be better if that was more acknowledged and communities of media more divided into multiples, as that would create more different cultures.