I can't help but think turning down the town was a mistake. It might be bad initially, but I think he could turn things around. Ultimately it should pay off pretty well.
The fact they even brought that up I think means he'll also come to that realization, or have some reason he needs it. Talking about raising armies, conquering territory, etc... yeah he's obviously going to move in that direction eventually. I don't think they're going to tease those concepts and never use them.