Yeah... He's supposed to be the MC of the story which in turn makes the world actually revolve around him, it's an effect based on self confidence, once he loses it, it vanishes... But people can either remain affected or get free according to their inner feelingsAbout the girls liking him, I read some spoilers in past comments, and if what I read is right
he have some sort of Harem protagonist power that makes girls attracted to him(he isn't aware of it), that is why the little sister still likes him even when he acts lamer than MC, from what I read the plot will take some meta directions in the future.
... Some characters like the sister can get free thank to her bond with her brother and eventually with shiho, or remain in love with him like the two other girls