- Joined
- Jan 24, 2018
- Messages
- 3,230
@Mr_Detective Are you me?
Complete Server Overhaul, New Affiliate, and Banner Update
- Thread starter Plykiya
- Start date
- Status
Member
- Joined
- Jan 28, 2018
- Messages
- 19
Nice, this fixed the thumbnail loading issue I've had in Tachiyomi since pretty much the day the MangaDex extension became a thing.
Dex-chan HATER
- Joined
- Mar 17, 2019
- Messages
- 17,017
mangaka are still putting masters/slaves in their fantasy mangoes in current year [insert current year]? yare yare daze, boya
Dex-chan lover
- Joined
- Dec 15, 2019
- Messages
- 3,188
girls should date girls yes but only if they are both guys hue
Active member
- Joined
- Sep 19, 2020
- Messages
- 620
Super straight guys should only date super straight girls.
- Joined
- Oct 31, 2020
- Messages
- 34
Nice! Everything loads faster now, thank you for this
- Joined
- Feb 16, 2020
- Messages
- 470
@nonamethanks: Thanks for your valuable input, here's a few pointers on things you might want to consider as for why the approach you mention is wildly off the mark
On the topic of your auction server suggestion:
- DB space maybe isn't as important as it being fast?
- Maybe compute resource on it is too?
- Maybe network speed and latency are too?
- Maybe fast DMCA-proof reverse proxies aren't free?
On reading comprehension:
The post reads > 2 API/slave db server
I think it's clear each of these does BOTH api and db server, which is quite different when it comes to compute requirements from your "a single slave server"
Finally
> People can (and do) host entire websites like Gelbooru for that cost alone.
1. a booru (esp. gel) isn't nearly as complex or popular as this, otherwise BW costs alone would have sent them to the moon
2. source: you
We shall (and truly do) keep an eye out to avoid useless spend as no-one wants that. Neither you nor us.
On the topic of your auction server suggestion:
- DB space maybe isn't as important as it being fast?
- Maybe compute resource on it is too?
- Maybe network speed and latency are too?
- Maybe fast DMCA-proof reverse proxies aren't free?
On reading comprehension:
The post reads > 2 API/slave db server
I think it's clear each of these does BOTH api and db server, which is quite different when it comes to compute requirements from your "a single slave server"
Finally
> People can (and do) host entire websites like Gelbooru for that cost alone.
1. a booru (esp. gel) isn't nearly as complex or popular as this, otherwise BW costs alone would have sent them to the moon
2. source: you
We shall (and truly do) keep an eye out to avoid useless spend as no-one wants that. Neither you nor us.
- Joined
- May 29, 2012
- Messages
- 594
@nonamethanks
That's clearly a 24TB HDD archive/storage server.
Our servers are expensive because we NEED to pay for lots of cores/threads, lots of RAM and NVME storage for all the processing, caching and moving data around without it crippling the site. You can't do that on cheap ass archive servers were you are paying for volume of HDD space - especially since at that price point it's backed up by geriatric CPUs, slow RAM and low public bandwidth.
Also, I can pretty much guarantee Gelbooru is not ran on just those kind of servers. I dunno if lozertuser posts his costs anywhere, I can't remember seeing exact figures (long time user of Gelbooru myself), but feel free to try and prove me wrong. He may use them for storing copies of his archive (which I also doubt, cos their public bandwidth is probably too low), as they are intended, but he is also going to have higher spec servers - that cost a lot more - to cope with all the searches, caching and database handling - not unlike us.
We only buy servers that make sense for the site to use. Buying cheaper ones at our current user level would make no sense, they wouldn't be able to keep up - see the past several months were our old servers had issues as demand was too much for them. This upgrade is good for us because we can now scale better horizontally and no longer need to vertically scale and at the same time it gives us some good headroom for growth before we will need to upgrade again.
TLDR - You don't know what you are talking about.
That's clearly a 24TB HDD archive/storage server.
Our servers are expensive because we NEED to pay for lots of cores/threads, lots of RAM and NVME storage for all the processing, caching and moving data around without it crippling the site. You can't do that on cheap ass archive servers were you are paying for volume of HDD space - especially since at that price point it's backed up by geriatric CPUs, slow RAM and low public bandwidth.
Also, I can pretty much guarantee Gelbooru is not ran on just those kind of servers. I dunno if lozertuser posts his costs anywhere, I can't remember seeing exact figures (long time user of Gelbooru myself), but feel free to try and prove me wrong. He may use them for storing copies of his archive (which I also doubt, cos their public bandwidth is probably too low), as they are intended, but he is also going to have higher spec servers - that cost a lot more - to cope with all the searches, caching and database handling - not unlike us.
We only buy servers that make sense for the site to use. Buying cheaper ones at our current user level would make no sense, they wouldn't be able to keep up - see the past several months were our old servers had issues as demand was too much for them. This upgrade is good for us because we can now scale better horizontally and no longer need to vertically scale and at the same time it gives us some good headroom for growth before we will need to upgrade again.
TLDR - You don't know what you are talking about.
- Joined
- Feb 16, 2020
- Messages
- 470
@nonamethanks jfyi, MD @ Home is pushing about 10PB per month. A cool ~27 times more than Danbooru
mangadex.network is a good place to look at this ; consider this is only images, and only ~35% of traffic is reported, due to apps not being compliant with reporting for example
mangadex.network is a good place to look at this ; consider this is only images, and only ~35% of traffic is reported, due to apps not being compliant with reporting for example
- Joined
- May 29, 2012
- Messages
- 594
@nonamethanks
Okay.
Compared.
So they do 377TB in a month.
We do a several PB a month.
Not even on the same scale. We dwarf their traffic. Which makes total sense when you consider how many images a chapter contains, how many chapters are hit every minute, versus people opening a few images at a time there. And that's STILL excluding all the database access, api hits, webserver hits, etc that require the high spec servers to handle. You seem to be confusing storage space/bandwidth with actual server usage and just how taxing on servers it is to process thousands of requests and database updates per second.
TLDR - You don't know what you are talking about.
Okay.
Compared.
So they do 377TB in a month.
We do a several PB a month.
Not even on the same scale. We dwarf their traffic. Which makes total sense when you consider how many images a chapter contains, how many chapters are hit every minute, versus people opening a few images at a time there. And that's STILL excluding all the database access, api hits, webserver hits, etc that require the high spec servers to handle. You seem to be confusing storage space/bandwidth with actual server usage and just how taxing on servers it is to process thousands of requests and database updates per second.
TLDR - You don't know what you are talking about.
- Joined
- May 29, 2012
- Messages
- 594
@nonamethanks
You're the one is keeps bringing up storage and bandwidth like it has some relevance on CPU/RAM usage for the scale we need. How does a 6TB NVME server help us with processing so many requests? I can tell you 100% for sure it will be dwarfed by the CPU/RAM availability we have on our new servers - that we actually NEED to keep the site going. Once again, storage space is not what keeps a site going.
If it weren't for MD@H we couldn't afford to run the site. That's why we are careful with our server buying and only pay for stuff we actually need.
TLDR - You don't know what you are talking about.
You're the one is keeps bringing up storage and bandwidth like it has some relevance on CPU/RAM usage for the scale we need. How does a 6TB NVME server help us with processing so many requests? I can tell you 100% for sure it will be dwarfed by the CPU/RAM availability we have on our new servers - that we actually NEED to keep the site going. Once again, storage space is not what keeps a site going.
If it weren't for MD@H we couldn't afford to run the site. That's why we are careful with our server buying and only pay for stuff we actually need.
TLDR - You don't know what you are talking about.
- Joined
- Jan 18, 2018
- Messages
- 2,064
@nonamethanks
>6TB
That fits less than half the regular archive.
>16TB
That fits the regular archive, but there's not enough spare for the data-saver archive and HDD arrays stopped being able to handle our outgoing requests more than two years ago.
That server also has significantly less total CPU power than what we use. We don't have Cloudflare to significantly cut down on requests reaching our infrastructure either.
>6TB
That fits less than half the regular archive.
>16TB
That fits the regular archive, but there's not enough spare for the data-saver archive and HDD arrays stopped being able to handle our outgoing requests more than two years ago.
That server also has significantly less total CPU power than what we use. We don't have Cloudflare to significantly cut down on requests reaching our infrastructure either.
- Joined
- Feb 16, 2020
- Messages
- 470
@nonamethanks That was merely to give you an idea of the scale difference... Don't be so voluntarily ignorant of the meaning of our replies.
> the absurd cost is driven by the fact that you use memcache instead of something better like redis
surely it's memcached's fault yeah lmao
What's next? MySQL too bad, we should be using MongoDB instead? Maybe also some kinda serverless arch? Event-driven setup?
Spouting names of (cool) techs whose tradeoffs you have no understanding of doesn't make you sound smart to the educated ones, you know
> the absurd cost is driven by the fact that you use memcache instead of something better like redis
surely it's memcached's fault yeah lmao
What's next? MySQL too bad, we should be using MongoDB instead? Maybe also some kinda serverless arch? Event-driven setup?
Spouting names of (cool) techs whose tradeoffs you have no understanding of doesn't make you sound smart to the educated ones, you know
Dex-chan lover
- Joined
- Jul 24, 2018
- Messages
- 654
@blurgh123
Let me tell you something AMAZINGLY obvious here. The master/slaves refer to computers, not actual human beings. Those are the most appropriate terms given the roles of those respective machines.
That's the most absurd "woke" nonsense. There is no real world impact from that, and ZERO benefit to changing it. There are real problems/injustices in the world, and this isn't one of them by any stretch of your fevered imagination. "Issues" like this are a laughing stock. And after you go "raising awareness" of such silly non-issues, no one is still listening when you raise an actually good point. And just why would you expect to be taken seriously after that? You deserved to be mocked.
Let me tell you something AMAZINGLY obvious here. The master/slaves refer to computers, not actual human beings. Those are the most appropriate terms given the roles of those respective machines.
- Status
Similar threads
- Locked
- Suggestion
