@Red225
I believe it's saying that the Duke would be alienated by adventurers, not the princes. It says in the chapter that Silver is well respected and shit so if the Duke made Silver look bad by turning him away, other adventurers would refuse to even bother with any future requests by the Duke. We haven't really seen enough of the world to really say, but I imagine adventurers are probably important and well respected in various ways so having adventurers essentially boycott your entire territory would probably be devastating (or at least extremely annoying).
The Prince is basically threatening not to mediate that. He's basically saying, "You guys fucked up by pissing off Silver, who we're buddies with enough that he'll listen to us when we ask. You guys better repay this offense to Silver (and by extension us) if you want us to talk to him and try to calm him down."
Or in other words, it doesn't matter if the princes alienate the Duke if the "unrelated" third party that are the adventurers take offense. It's a cats paw, essentially.