Honestly, I think this man can use his reputation a lot more to build his kingdom. I know he's kept his power a secret for quite a while, but I think it's a good time to reveal it. It's practically the same as "Appraisal Aristocrat" but difference is that MC is a kid in that manga, so it is in his best interest to keep it a secret until he can secure his safety. But here, MC is at the least above the average adventurer, and in the worst situation could still be backed up by his former acquaintances. Reminder, they don't hate him at all, the split came down to a very reasonable understanding from both sides, and he was even compensated with a whole fiefdom, albeit one in poor condition, but he definitely has potential for greatness as a leader.
So if he used his reputation, he can very easily attract hundreds, or even thousands of people coming to help the person that personally trained, mentored and grew the famous hero party. Neither does it need to even be infantry or adventurers, but bureaucrats, more merchants, lords, nobles, and some of the most efficient farmers. As long as the world understands that his power isn't limited to training just adventurers, but even recognizing the true talents of any person.