TBF he was shocked and hurt, believing that all this time he had been annoying them and just tolerated. He felt that his CHFs Ellie and Roland (and the "others", but to this point we haven't seen any "others") had betrayed him by pretending to be all friendly with him while having already decided to expel him from the party. Ellie's
friendly comment about lunch seemed fake and led to his "
I've been taken care of" (already fired) with an implied "so you can stop acting".
It's easy to criticize his not asking but due to his state of mind it's also easy to understand why he didn't stop to talk. He just wanted to get away from the den of deceit where he no longer belonged.