didnt they say he was just average l ol
No, it was slightly implied (so that this would be a "twist") but it always seemed pretty likely that wasn't actually the case, so this development's pretty expected. All that was actually said was that he "failed to meet expectations", right after we see him having a nightmare about what was done to the demons + crying over the destruction of their homes.
Taking it at direct face value that can be construed as "he was weak", but considering he was summoned as a hero, whose typical role is to subjugate the demon race, and then went on to sympathize with the demons instead (to the point where he later held classes for them), it's clear why he "failed to meet expectations" of the kingdom and then got booted from his role.