Doesn't sound like a programming issue.
It feels that way cuz light mode is easier for reading text especially if you have visual impairments. More light makes our pupils contract which let's us see more depth and stuff.
When reading e-books and articles I prefer light mode.
Dark mode is comfy + saves battery.
Either that or you're using a site which is really fast because it lacks some features.
I've never noticed that before, as I usually only use one mode for one website and don't switch between them.
What I'm certain of is that it isn't a programming issue. It's probably just a matter of the display and our eyes.