@Psychronia
I see.
The problem is you don't understand what the medieval/middle earth bandits (deserted soldiers) actually are.
(Please take a little of your time to read GATE: Thus the JSDF fought here to see)
They rob any merchant carriage they see on the way.
They rape any women to satisfy their needs.
After gang-bang them until their mind is broken, the bandits will sell them as slaves (illegal though)
(If you say it's another series and unrelated to this series then please take a little of your time to read the earlier chapters about the war.
What did the empire troops do before they desert?
Kill innocent villagers and feed them to monsters.
It's just recently so the citizen of the kingdom don't forget it that easily.
Everyone in the carriage remembers it well)
Basically:
The farmer wants to protect his wife and daughter from being captured and sold by the bandits so he says "he will fight"
The old man sees that and says things like "kids should be with their family instead of being captured and sold, even he is a bad person in the past, but at the end of his life he wants to help someone for once)
The merchant doesn't get any profit but he says "I will fight as well because he can't stand seeing women get captured and sold"
TL;DR
That selfish man isn't stupid to say out loud "Hey, just hand all the luggage and women to the bandits to save us, men's skin"
He omits the "women" part, which everyone in that carriage understands it without saying.
After this, everyone looks at him like trash.
PS: I didn't make anything up.