Eh, I'm pretty sure that people are mixing up the (US) First Amendment and free speech in general. Mangadex is a privately owned website and as the First Amendment generally applies to public places only, Mangadex cannot violate it no matter what they do. However, not violating the First Amendment is not the same as not restricting someone's free speech. Furthermore, in US legal law, generally, hate speech is covered under free speech laws. Only outright obscene speech is not considered free speech or things like lying or obvious death threats.
One reason for hate speech not being legally restricted is that it's hopelessly vague. Is the statement "I hate people of color and hope that they burn" hate speech? What about the phrase "I hate white people and hope that they burn"? Assuming all people are equal, both statements must be either equally hate speech or not hate speech. However, only the former is generally considered offensive. As a minority, I find this laughably inconsistent.
Basically, Mangadex did restrict free speech, but it's their right to do so is my position. Ultimately, it's up to the actions of people offended to either act upon this or to not act upon this.