I mean, yeah it's rough that he was kicked out, but he really didn't do himself any favors by refusing to explain the situation (as he himself acknowledges). Though he has his reasons.
I'm not saying the party made the right choice, but it wasn't unreasonable for them to think he was holding them back.
I'm not saying the party made the right choice, but it wasn't unreasonable for them to think he was holding them back.