Get out your tinfoil hats and take a look at my corkboard and scribblings, because I've got a crazy, crackpot theory after the timeline of this last chapter. Feel free to tell me I'm full of hogwash, because honestly, I'm flying this one by the flimsiest of pretexts.
We can assume Yoshida is basic salaryman age, say early thirties. If he is explicitly stated to be younger than this whole thing falls apart, but I'll go forward on the assumption that that age is accurate for our wild theory-crafting purposes.
What if Sayu is, by cosmic coincidence and law of conservation of drama, Yoshida's daughter borne to his senpai/ex from high school? Yoshida's high-school ex/senpai was explicitly having sex with him, presumably without contraception, right up until her graduation. His memory of her is that "she seemed nominally secretive" in addition to being "bright and cheerful," which is not unlike Sayu herself. If Yoshida is about double Sayu's age (32 or 33), then the timeframe does work out. Of course, this assumes that Yoshida's high-school ex/senpai broke contact with him partly to keep him from being bound by the responsibility of being a teenage father, further, the conception would have had to have happened right as ex/senpai graduated, which is unlikely but possible, given how I understand Yoshida describing their continued sexual activity. If we are to trust Yoshida's characterization of his ex/senpai, then hiding information because she cares about him would be in character. Further, if we to trust Yoshida's perspective of ex/senpai possessing sexual experience, her silence about her past sexual history, wether it is abuse or consensual, is really considerate of her to conduct her sexual relationship with Yoshida without putting any serious expectations on him, to let come into his own without judgement. It is possible that the trope of parents moving because of work, coupled with the need to care for ex/senpai's child caused them to move to Hokkaido(?) or wherever Sayu is from.
If this ends up being the case, the series can swerve into Yoshida having strong bond paternal love for Sayu, even if he stays broken up with his high school ex/senpai, though I'll admit that's mostly wishful thinking on my part. He can still have her as a part of his life in the non-romantic and non-sexual sense as he's been trying to for most of the series. Though I don't know if that's where the author is taking the story at this point. Or even if Yoshida isn't Sayu's father, Sayu's mother could still be the ex/senpai Yoshida detailed in this chapter; I can imagine a teen mother's daughter in particular having an imperfect home life, regardless of who the father is.
TL;DR: Could Yoshida be Sayu's dad?! It would be interesting if.