Author-sensei can easily sneak in a flashback that highlights that Baku totally retrieved the burning handkerchief for sentimental value or w/e, and highlight that once he'd won but before this began, Baku deduced that they would play a crucial role. I'd wonder why Yakou bothered to get a second No. 0 cloth while he was on the island (not to mention how) but I think any old explanation could work for me.
But it would make even more sense if Yakou can do more good outside of the refereeing aspect and so sided with Baku in his own way by forfeiting.
-shrug- I ought to give up with this manga since I'm always wrong lol