It is perfectly fair to rate a manga poorly even if you have never read any more than the title, synopsis, and saw the cover image. If you rate sonething poorly in this situation you are disagreeing with something you saw there. I personally dislike reading yaoi manga but often times I will open one because the title and cover are deceptive. People giving yaoi stories bad ratings without reading them (although, how the hell do you know they aren't giving the story a chance before rating it) might be people annoyed by this deception.
It is also fair to hate the very existence of something(or at least the existence of its support/communities). This is extremely common for "too" popular movies and videogames. For example, tons of people hate things like The Transformers franchise or the Marvel cinematic universe yet have only ever seen one movie or, oftentimes, none. The same is true for games like Assassin's Creed or Call of Duty. This doesn't mean their opinions are invalid though. They just aren't trying to objectivly review the quality of the art (which is impossible to do anyway). What they are rather doing is saying that this should not exist or, at the least, should not be this highly rated.
(Also, I hate the idea that the mangas that I hate enough to give a low star rating to should follow me around. Personally, I rather want it to be that when I give a mangs a one star rating that it be immediadiately deleted from my history and blocked so that I never have to see it again.)