In the early chapters I really could not entirely understand Joe's distaste for Youko. I kind of got where he was coming from, but mostly felt he was being irrational and unreasonable.
At this point though (if not a bit earlier), I can really see where he is coming from distrusting and even hating Youko. She has repeatedly manipulated, controlled, interfered, and played games with Joe and others (e.g. Takigawa, the boxer who was tricked into facing Himaru, and Himaru himself) in order to get what she wanted. Even to the extent she may have good intentions, she also clearly has selfish intentions (e.g. getting Joe's wildness back) and is willing to step on others to achieve both. Indeed, even well-intentioned deceitfulness is still reason for others to distrust you.
Plus, she herself puts on a mask of cold-heartedness to others, especially Joe. It's no wonder that he does not want anything to do with her by this point.