I have a theory that the real reason children are gemstones to him is because he's close to not needing the rock filter for them.
The reason he has the rock filter in the first place is because he was afraid of getting hurt by getting too close to them. The mc pointed out how he doesn't really have to worry about that when it comes to kids, so there's no reason for them to have a filter.
Maybe once he gets close enough he'll see other girls as gemstones and eventually see them normally once he's used to them.