The instant something doesn't go his way (or even seems to not go his way) Kaiji is ready to yell, "cheats!" and throw out all he thinks he knows about a person.
That's honestly no different than his friends betraying him so easily based on rumors and speculation, or all the misunderstandings between Mitsuyama, Chang, and Mario. Ultimately, either you trust the person in front of you and what you know about them, or you don't.
Don't get me wrong, I get why Kaiji (or anyone) would be worried / suspicious, but that just further proves Kazuya's point about human nature. Also worth looking back to the E-Card gamble when Kaiji realized that what he was seeing in Tokugawa (and vice versa) was really a case of looking in the mirror.