Because "that" is the only thing that could be seen as cowardice from our little MC's behaviour so far, as inane as that notion was. Him just returning the book immediately is not a show of bravery. It's idiotic.
Not immediately returning that bombshell of a book is perfectly understandable. You have to consider the culture of this manga. It's Japan, where social reputation is pretty mighty important. That's why he didn't just give the book back to the girl in front of everyone, since otaku culture is seen as a pretty lowly thing.
Should he returned the book to her right then and there, he risk putting down her social reputation. And more importantly, his own. People would gossip about why he returned the book in front of everyone. People might accuse him of doing it intentionally by stealing it from her bag to ridicule her, and similar hypothetical situations. The truth doesn't matter to people, what they want is spicy stories, and spicy stories are what they will project.
That part of Japan's culture is fucking dumb, but it is how it is. I heard that the newer generations are not following it as hard as the previous ones. Anyway, my place has a similar kinda thing going on as well (though not as intense as JP), so I can easily understand why the MC was all hesitant.
