Because the chance of destiny cannot be counted on a simple fortune. Ken here only alone and Nobunaga entrusted him with the fate of 3 clans, either Uesugi decided to converse with Nobunaga or what Katsuyori beliefs in the destruction of Oda. Either way, Ken alone hold a slimmer chance.