We're only six chapters in, and Hayami fully called her "incredible" earlier. No moron worth the title is going to catch on any faster than that.
Honestly at this point I'm rooting as much for "Ayumu makes some friends" as I am for any romance.
It strikes me that there's another reason for Aizawa to seem so different both as a ghost and in pre-idol flashbacks to everyone's perception of her idol self: if Aizawa wasn't driving the car for long periods while she was an idol. Nagi (or the theoretical ghost behind the ghost?) could have...