Hol' up, Shiori turned him down in middle school? Is he talking about when she asked him to keep being her friend in ch. 4 just before she transferred away?
In ch. 4 he says he changed his mind and didn't confess after hearing her ask him to keep being friends... so how would he get turned down if he couldn't confess at all?
Unless he somehow took it as a rejection without confessing... if he did manage to confess and she did turn him down though, that's a game changer.