I kinda agree with Mika here. Of course, that doesn't mean he should bend over to Noa's every whim, but c'mon. He knows that's her way of showing affection and how much she treasures their relationships, so always trying to mine their time together when he knows how important that is to her is...
The thing with Noa is that she loves open gestures of affection, that's why she is okay with hugging. But a gesture so subtle and more secretive as handholding feels just too intimate.
...like, i know they definitely didn't go all the way because harem manga bullshit and everything but...it's simply absurd for me to imagine they didn't fuck in these scenes...