Well... he clearly did that after Julchalga said what those women were being done to... I don't particularly mind if that is his trigger, just shows that he has a good heart, if a little fragile at some point.
But I won't fault Bard if he decides to keep an eye on Ven from now on, triggered by a good reason or not... he did just jump into a disadvantageous situation and dragged his companions in his frenzy, that isn't a thing you just brush off.