Because that's telling the girl you like that you went on a date with another girl
He didn't really see it as a date, only Lemon did. But it's still him going out all day with another girl, just the two of them. Not just another girl, but a girl Natsumi and Sunao are both well acquainted with, that he has a complicated history of feelings with, and a girl he's in the body of half the time. So even if, to him, it wasn't a date, he likes Natsumi and is prone to overthinking things, so it's only natural he wouldn't want to tell her.
Really, this is a giant mess and Natsumi deserves better than to be dragged around by Sunao and Lemon like this.