Probably because Yamada isn't a serious character while Asou is a character we are meant to take seriously, it also doesn't help that he's an 11th hour rival which is probably the least likable character in romcoms in general.
Also Asou is just... Too normal. To the point other characters are pointing it out. It's ringing alarm bells in our heads that there is something really wrong with him.