What makes it not creepy is if the loli/shota is the aggressor with the precocious crush, and the older person actively worries about their wellbeing, doesn't abuse the age difference in terms of power and knowledge, and doesn't reciprocate until they're adults (or at all), and also doesn't happen to be their surrogate parent (Looking at you, Usagi Drop). It's a matter of perspective. As readers, we're close enough to be able to tell that the fortune teller lady is perfectly harmless. In real life, we almost never get to look at a relationship that closely.
Gender is, or should be, irrelevant. Though I'm slightly more forgiving towards same-sex relationships because there's no danger of an unplanned pregnancy... I'm not sure if that's a creepy thought to have or not.