The main characters specifically moved their camp to a hard-to-see location as far from the other group as possible, and it was established that there was bigger game like deer in the opposite direction, which is why the other group didn't wander close to the MCs in search of food. The MCs are mostly catching fish and rabbits with poles and traps (a strategy that's harder to figure out without prior knowledge), whereas the huge group would want to use their numbers to their advantage hunting big animals with spears. Of course, larger animals have smaller populations, so that hunting style can't sustain a large group as long, so the hunters have probably been search farther and farther away from the main characters.
