@Veshv There are two options:
1. The referees can directly challenge number 0.
2. The referees have to work their way up to number 0 by challenging referees 1,2,3...
Based on the events in the abandoned mine, it is most likely option 1. In that case, there is only real reason to challenge referee 0, challenging other referees would be just to prove themselves like Kyara, or to kill boredom, or personal reasons like Mekama. Everyone knows that Nowa is a weak old man in wheelchair, so there is no reason for the others to challenge him.
If it's option 2, they would have to defeat Yakou in 10 seconds to get to Nowa so yeah... Good luck with that.