Tadano probably think he is too ordinary that he doesn’t deserve to be loved by Komi. He may be thinking that Komi befriended him thank to her disorder or because Komi pity him.
Thing is beside Najimi I don’t see Tadano hang out with anyone else.
He must be having a very low self-esteem after that dark past.