He's hinted that there are reasons for being the way he is, but he wasn't ready to talk about it with the girls. Probably something traumatic from when he fought the demon king.
The problem is, it'd have to be something really major to have this kind of effect on him. I suspect the author has written himself into a corner here, 'cause there's not much that could explain his extremely low self-esteem.
That's just about his general attitude and work performance, though. I don't consider his inaction toward the girls as being spineless. He's a teacher and he's almost 50 years old. I'm almost 50 years old and, while not a teacher, I have subordinates. If a young subordinate started coming on to me, I'd probably respond in a similar manner to Washio.