@Salfaro: Probably the same reason he didn't panic when Aries save them: He didn't recognize them by name, but by form (after all people might have same/similar name by coincidence)
The people living now likely don't know that all the 12 stars have human forms.
That's why he only panic AFTER seeing Aries's ram form.
In case of Virgo, they probably only know of her grandma's looks, thus doesn't realize she is the NEW Virgo.
On the chapter: Oh boy, Sei is about to get introduced to his crush's family....this is like the romcom moment where the guy finds out his waifu is yakuza boss's daughter, and they're not amused by this new wimp hanging out with the ojou.