Everyone going 'it's not racist!' is ignoring the subtext involved, where 'the paler the skin, the better' is a severe problem in some countries, leading to people even artificially whitening their skin. Sure, IN-UNIVERSE there's excuses. But the context is important. If I take someone who is coded Jewish, and then proceed to paint them the villain, 'but it's just fiction' isn't an excuse.
So you can do two things. The first is investigating both your own prejudices and why people dislike 'X-coded is used for villains' and 'white = better' tropes, and actually listen to them. The other is insisting that it's perfectly fine to keep throwing harmful stereotypes around. If you do the second, whelp, congrats, you're a bigot in denial.