Brown is a rare eye color?
BROWN?!?
I get that it's a fantasy world, but they drink black tea which doesn't seem to be cultivated locally, and the fact that only the upper class individuals seem to consume it would point to it being a luxury import. Additionally the world shares all of our herbs and spices therefore it must be similar to our world in more ways than one. So I would think that people with brown-colored eyes would be the majority like in our world.
Black hair isn't that uncommon for Europeans either, but it's a common trope for isekai characters to be reviled/stand out due to their "super rare" black hair so I'm used to that, but the eye color thing is just another layer.
Sorry for my rant.