Remember, Tadano doesn't think he really suits Komi, and probably not Manbagi either. He knows he's painfully average and thinks they're both way above his station, so he's forcing himself not to have romantic feelings since he thinks he'd just get rejected anyway.
Kawai-san in middle school really broke his spirit.