So uhh. I guess this is fine because Asami did what Yoshida just probably forced himself not to.
I mean, personally I feel that just gives Yoshida more plot to himself. Not romantically, but the possibility that he's believing himself to be weird. With everyone else hammering him the idea that his type of kindness is even unreal.