Maybe they should add weighting to the rating. If somebody has an unusual high count of "1", "2", or "10"-ratings in his overall ratings, another of those ratings is weighted less (that guy only counts as half a person or 1/10). It's enough to keep track of the number of his overall ratings without linking it to the mangas and to set the weight only in the moment that guy rates something. The value of the weight only gets additionally stored in the users history, so it can be undone, when a rating is retracted by the user. That way the ratings don't have to be reevaluated every day. The opinions of people, who try to rate fairly are "louder", and dweebs, who only rate "1" or "10" aren't taken seriously anymore.
@asdgfh: People are fuckups. They rate stuff as shit (1) or the best thing EVAR (10), because they get annoyed or hooked too fast. What konzolmester said in his post.