The last thing Shirogane's mom said to him before leaving was, "People don't always stay in love"? No wonder why he gets to paranoid when it comes to Shinomiya.
I feel like a lot of people are forgetting that the experience that made Tadano be able to read the mood so well in the first place is the same experience that made him think girls wouldn't be that into him.