Riku was there, in the same house, all the time when Sora was still in the womb. In that sense before being born, Sora did spend more time with Riku than Nanami did, because at the end of the day, Riku's mom and Nanami's mom obviously lived in different houses.