I may be in the minority then for liking this pair. Here's my take.
I view manga reading as escapism. When I read manga, I know that this is and will always be 2D, and I can compartmentalize. Manga is manga and IRL is IRL. IRL, I detest child molestation, grooming, etc. But when I read manga, I can appreciate the couple for how the mangaka presents it to me: a comical take on an adult male crushing on a little girl, and in the process, turning the tables on power and relationship dynamics. An adult male being fully subservient to a cute little girl-- that subverts expectations and is therefore an interesting story to tell!
You can tell the mangaka is aware that this is a taboo, of course. It's shown through the disapproval of the elder brother, the butler's own self-control to wait for 11 years, etc. I actually like how the mangaka is delicately handling this "romance" (quotes will be removed after 11 years in the universe

). Should it go beyond the innocent "crushing" part while the kid's still 7, I'll definitely be disappointed because it'll ruin the theme of their relationship. But I won't judge it from the IRL lens. Similarly, it's how I feel about people who use tentacle rape as their sexual fantasy stimulant, but IRL are functional, competent, sane people who won't ever try to do it with an octopus.