This would be a lot more entertaining i the protagonist wasn't a GIGANTIC asshole who blames his isolation on choosing to dislike other people because they're the problem.
He doesn't deserve sexy librarian fun. Or sexy book-girl fun. Or any fun at all.
And this author has shown they can work in that direction too. Mizuno from Scary Girl, Cute Girl is rough around the edges and rude/prickly to people, but is also generally a nice person who internalizes everything bad and blames herself even when it's unwarranted. This guy's just a prick who makes it seem like anything positive he does is totally by accident.