This manga is made by Oogami. She's a really unique main character and is incredibly endearing. She knows what she wants, she's horny as fuck, nothing gets her down for too long (when she was depressed it literally took two chapters for her to get over it) and when presented with some bullshit like this from Yaginuma her reaction is "fuck that!" She's just a really great girl you want to see succeed.
Yaginuma is probably thinking that relationship stuff will get in the way of friendship and him learning how to...well, be a person. Which is why he's trying to head that shit off at the pass because he never had a romantic relationship with anyone as her goal. Obviously a normal person (well, maybe not in Japan) would just go for it and figure out the rest along the way but since he's very cold and analytical about stuff he saw the best way to reach his stated goal was to create some distance. Even beyond that from his conversation before he really doesn't understand romance either. Getting into that may be way over his head at this point as he'd now be contending with two things he doesn't know how to do.
Obviously it's a bit silly, but this is a manga. If he talked it out and thought about it or had realistic reactions there wouldn't be any conflict, tension or drama to carry it along. Maybe I just don't care in this case because of Oogami's reaction so she's going on the attack immediately and other manga have done so much worse.