"You didn't even have any dark or evil intent."
I'm sorry, but villains don't tend to see themselves as 'evil'. I'd find it more realistic if he said 'you were too soft to make hard decisions' because then it would make sense, but no-one calls themselves or their side evil. Maybe they'll say 'we commit the necessary evil to achieve our goals' but everyone, even antagonists, will see themselves as doing the right thing. It's just that people have a different definition of what is right.