Well. They did go after other people so Doppo would want a piece. Baki would want on since it would be a way to train and get ready for his dad. Old glasses dude is probably PISSED at how he got beaten. Same with Retsu being pissed. And Hanayama is there for a good time. XD
Plus, I think the implication is that most or all of them wouldn't quite be able to give a satisfying defeat to each other. At best one of the convicts would win and then he'd keep on rampaging. At worst it'd be a couple draws depending on the match ups and if they even consider a non surprise meet up to brawl like this a worthwhile fight.
And above all. It's Baki logic, shut up. XD