Bigger isn't necessarily always better. While it is nice to have a wider spread of abilities, you have to factor in the cost of a larger party. It takes much more time and money to get them up to speed, and, more pressingly, they sponge up experience like there's no tomorrow. In the time it takes to make a proper full party, you can oftentimes make more progress with a small team of elites that level up quickly and get all the best gear. Neither plan is inherently better than the other, but there are drawbacks to both.