Ah, no, he actually saw Kodaka as a kindred spirit, not Yoshino. In the present day he told Tsubaki that he knew he wasn't worthy of Yoshino but went to see her because he was curious and the only way to see her was to sneak off because, like it says in this chapter, he's not a yakuza's grandkid. :)