Iirc this was mentioned in the manga too in throwaway lines. Before seeing these women, iirc it was Anzu that mentioned them being... victims of the previous mayor.
And it was mentioned that the previous mayor and the bandits... "used" captured women while killing all the men, you connect the dots...
Lot to be said about the progression from LN to Manga to Anime though - in general, not just KKKBear. Usually, the LN or WN will have some pretty dark bits, the manga will gloss over them, and the anime will replace them with fluff.
Case in point here; anime just has a framing shot then a distant shot with a bear-shaped cloud; manga just has an attack but doesn't really go into detail on the captives, while the LN has this:
"We helped the captured women off the carriage. They cried together. Hugged each other. I could imagine what they’d been through, but I didn’t really know what I could say to them. On top of the bandits’ treatment, there should have been the other people they left the seaport with—husbands, parents, kids. And there weren’t any. Not anymore. So I said nothing, even as they went on thanking me."