@phil777 If I remember correctly, it's because of how they were introduced. Basically, when panties first came to Japan, they were a luxury good, meaning only high class ladies wore them. Because of this, people started fetishizing the rarity, and it just became ingrained into their society over the generations, despite them becoming more and more common. I guess you can call it a cultural fetish.
Edit: I should probably clarify that "high class ladies" refers to geisha and prostitutes, too, so that likely helped associate them with sex.