I'm a bit confused about why the oath was freezing them here. I recall them freezing up when the oath was made but I thought that was a one time thing due to the establishment of the oath. I mean, all that happened was they got a contract to work for him and they didn't mention any of the details, which I assume were about protecting his privacy, so how is just asking what are you doing anything against the oath? They basically asked again behind closed windows and there was no freezing then...
The first reason is who officiated the contract. It was a greater spirit, one who they considered to be a god.
Next is the strength of the contract they signed. Jean by now has contracted thousands of spirits of all types, including a fallen greater spirit. So any contract he signs has a vast ammount of power behind it. I forget all the stipulations of said contract, but as partial spirits themselves, the contract they signed has a greater effect on them then it would a normal human.
So a lot of what has them on edge is that their new master Jean is exhibiting some very concerning abilities. Not only is he strong enough to call down a God to officiate over a contract, but now he is appearing before them at their destination when they themselves used a non-standard method of travel.
And Jean does so with an utterly carefree attitude.