It's true that mature people don't provoke needless fights and know how to pick their battles, but they can also assess the situation and stand their ground when necessary. He is panicing pointlessly and were it not for the supporting characters, might even give in and become the princes' lackey. Meanwhile the prince himself currently has no way to touch him - his ships cannot be sunk or borded without an invitation and no one even knows his identity, plus the ships themselves can change appearence freely. He can sit tight and try to wait the prince out, or vanish without a trace the moment he's bored of it. There is no need to feel threatened or even threaten the country. It's enough to make them understand that he either cooperates on his terms, or not at all.It’s precisely because he’s an adult that he tries to avoid unnecessary trouble. Only a 12 year old decides things on impulse and is quick to throw fists. This dude is just an ordinary person thrown into a completely different world. I doubt highly that you would know how to deal with a member of royalty that can bring the power of a country to a fight.