My favourite time of the week has arrived!
I don't believe this. The opening chapter said that Gomi reads a lot of rape hentai, and he goes to soaplands and that kind of thing. He shouldn't be this hesitant when he's drunk and she's initiating physical contact. His reaction would make sense if he were a teenage boy or a total virgin , but not as an adult man who has spent time feeling female flesh and fantasising about raping women, especially when he's drunk and horny from smelling her body and she's just touched him and he's received nothing but positive signs. That should easily overcome his normal inhibitions and doubt.
I was expecting him to full on grab her thigh.
I don't think you are considering, or at least giving enough importance to, the chapter where he speaks of his ideals and views on romance and women. In the past, he made a decision for himself to never do anything to a woman because he doesn't want them to destroy his life. To be harmless in society (ie only goon to fantasy), 無害 in Japanese (a term that is often used on men who will never do anything sexual to a woman, meaning they're "safe" men). That is his big inhibitor and why he has not done a single thing to Kuroko despite the positive affirmation she has given him.
Notice how every bit of narration and internal dialogue (when it's not just being horny) is always worded in a cautious tone. He calculates his actions in minute detail not because he's trying to get in her pants, but because he does not want Kuroko to hate him. He doesn't want to disrespect her or do anything wrong, because ultimately he's more anxious/afraid than anything, no matter how unimaginably horny he is deep down. He has panicked over this multiple times now.
The door bursting open in this chapter metaphorically symbolizes him finally taking a step beyond that (or more precisely, the emotions finally being strong enough to tear those shackles whether he wants to or not), yet at the same time he is still doing everything he can to keep it together, hesitating, because of his huge fear of women. So him touching her finger was literally all he could muster in his jumbled emotional state after stepping out of his shell for the first time ever.
Him fantasizing about rape and having been with prostitutes does not at all undermine this because those were transactional relationships (or fictional ones) where he could feel comfortable he wasn't breaking social norms. Here it's entirely different, and it's also why he still calls himself a virgin.