Assuming Daru's former party was a lot ahead of what Athena is capable of now, it's necessary for Daru to be much stronger than he has appeared so far. However, if he had used his full power when adventuring with Athena's party, then Athena and the others would have had little to do facing opponents of their own level of even slightly above, and Daru would have always kept them safe. Then when they had ultimately clashed with a much stronger opponent, they would have been dead meat, not being able to keep up with Daru and the enemy, yet at the same time Daru would have been unable to protect them. For Athena & gang to grow stronger, Daru needed to let them fight enemies relying on limited support.
That being said, Daru was something of a bastard, until very recently. He wasn't really actively trying to make sure Athena & the rest actually do get stronger. He was content with watching Athena struggle. If he wasn't a bastard, I reckon he might have actually noticed, before long, that Est wasn't using his true potential at all.