He want to stick to give it all in fashion, it's valid.
Doing that, he fear that he can't give enough attention to a gf, ok.
But I'm mad about the author. He made Saotome think that he has been manipulated in the end again (poor Kujaku, she doesn't deserve that treatment) after 200+chapter.