If you think about, the other brothers' time away from the capital must have been entirely different from Yang Kai's. They had no cultivation problems and had received all cultivation aid from the Yang family that money could buy even before leaving. Since Yang Kai supposedly couldn't cultivate pretty much at all, he likely only received anything from his parents, whose influence was already quite small. Plus what use would it have been anyway, since he couldn't cultivate? So, the other brothers most likely made it to be inner/core disciples and had lots of buddies and probably underlings. They would continuously socialise with other elite disciples. Yang Kai was the lowest of the low, a trial disciple and the famous punching bag in the sect. He never developed the kind of personality one would need to have to sit in a private room of a high class inn and have fun with peers from elite families.