I agree, but I think both of his co-workers already know/suspect Ayano has a crush on him. And going by the little clues the flashbacks gave us, it seems he was too close with his previous client (I can't tell if it was a romantic relationship or not, but it probably was)