- Joined
- Jan 31, 2021
- Messages
- 28
Thanks for your hard work guys!
Complete Server Overhaul, New Affiliate, and Banner Update
- Thread starter Plykiya
- Start date
- Status
- Joined
- Jan 24, 2018
- Messages
- 3,231
@Mr_Detective Are you me?
Member
- Joined
- Jan 28, 2018
- Messages
- 19
Nice, this fixed the thumbnail loading issue I've had in Tachiyomi since pretty much the day the MangaDex extension became a thing.
Dex-chan lover
- Joined
- Mar 17, 2019
- Messages
- 9,695
mangaka are still putting masters/slaves in their fantasy mangoes in current year [insert current year]? yare yare daze, boya
Dex-chan lover
- Joined
- Dec 15, 2019
- Messages
- 3,189
girls should date girls yes but only if they are both guys hue
Member
- Joined
- Sep 19, 2020
- Messages
- 620
Super straight guys should only date super straight girls.
- Joined
- Oct 31, 2020
- Messages
- 34
Nice! Everything loads faster now, thank you for this
- Joined
- May 1, 2020
- Messages
- 5
Gonna be honest, I don't understand how you people are paying $250 for a single slave server, when 24TB auction Hetzner servers go for $55. Just put a reverse proxy on top of it if you're scared about takedowns, especially since it's just a slave. Even if you don't want the hosting to be in Germany or another DMCA-receptive country, $250 is just a ludicrous amount. People can
(and do) host entire websites like Gelbooru for that cost alone.
But looking at the history of manga hosters, I guess it's just a pathological trait of this kind of dev/sysadmin to just be bad with money.
(and do) host entire websites like Gelbooru for that cost alone.
But looking at the history of manga hosters, I guess it's just a pathological trait of this kind of dev/sysadmin to just be bad with money.
- Joined
- Feb 16, 2020
- Messages
- 462
@nonamethanks: Thanks for your valuable input, here's a few pointers on things you might want to consider as for why the approach you mention is wildly off the mark
On the topic of your auction server suggestion:
- DB space maybe isn't as important as it being fast?
- Maybe compute resource on it is too?
- Maybe network speed and latency are too?
- Maybe fast DMCA-proof reverse proxies aren't free?
On reading comprehension:
The post reads > 2 API/slave db server
I think it's clear each of these does BOTH api and db server, which is quite different when it comes to compute requirements from your "a single slave server"
Finally
> People can (and do) host entire websites like Gelbooru for that cost alone.
1. a booru (esp. gel) isn't nearly as complex or popular as this, otherwise BW costs alone would have sent them to the moon
2. source: you
We shall (and truly do) keep an eye out to avoid useless spend as no-one wants that. Neither you nor us.
On the topic of your auction server suggestion:
- DB space maybe isn't as important as it being fast?
- Maybe compute resource on it is too?
- Maybe network speed and latency are too?
- Maybe fast DMCA-proof reverse proxies aren't free?
On reading comprehension:
The post reads > 2 API/slave db server
I think it's clear each of these does BOTH api and db server, which is quite different when it comes to compute requirements from your "a single slave server"
Finally
> People can (and do) host entire websites like Gelbooru for that cost alone.
1. a booru (esp. gel) isn't nearly as complex or popular as this, otherwise BW costs alone would have sent them to the moon
2. source: you
We shall (and truly do) keep an eye out to avoid useless spend as no-one wants that. Neither you nor us.
- Joined
- May 29, 2012
- Messages
- 594
@nonamethanks
That's clearly a 24TB HDD archive/storage server.
Our servers are expensive because we NEED to pay for lots of cores/threads, lots of RAM and NVME storage for all the processing, caching and moving data around without it crippling the site. You can't do that on cheap ass archive servers were you are paying for volume of HDD space - especially since at that price point it's backed up by geriatric CPUs, slow RAM and low public bandwidth.
Also, I can pretty much guarantee Gelbooru is not ran on just those kind of servers. I dunno if lozertuser posts his costs anywhere, I can't remember seeing exact figures (long time user of Gelbooru myself), but feel free to try and prove me wrong. He may use them for storing copies of his archive (which I also doubt, cos their public bandwidth is probably too low), as they are intended, but he is also going to have higher spec servers - that cost a lot more - to cope with all the searches, caching and database handling - not unlike us.
We only buy servers that make sense for the site to use. Buying cheaper ones at our current user level would make no sense, they wouldn't be able to keep up - see the past several months were our old servers had issues as demand was too much for them. This upgrade is good for us because we can now scale better horizontally and no longer need to vertically scale and at the same time it gives us some good headroom for growth before we will need to upgrade again.
TLDR - You don't know what you are talking about.
That's clearly a 24TB HDD archive/storage server.
Our servers are expensive because we NEED to pay for lots of cores/threads, lots of RAM and NVME storage for all the processing, caching and moving data around without it crippling the site. You can't do that on cheap ass archive servers were you are paying for volume of HDD space - especially since at that price point it's backed up by geriatric CPUs, slow RAM and low public bandwidth.
Also, I can pretty much guarantee Gelbooru is not ran on just those kind of servers. I dunno if lozertuser posts his costs anywhere, I can't remember seeing exact figures (long time user of Gelbooru myself), but feel free to try and prove me wrong. He may use them for storing copies of his archive (which I also doubt, cos their public bandwidth is probably too low), as they are intended, but he is also going to have higher spec servers - that cost a lot more - to cope with all the searches, caching and database handling - not unlike us.
We only buy servers that make sense for the site to use. Buying cheaper ones at our current user level would make no sense, they wouldn't be able to keep up - see the past several months were our old servers had issues as demand was too much for them. This upgrade is good for us because we can now scale better horizontally and no longer need to vertically scale and at the same time it gives us some good headroom for growth before we will need to upgrade again.
TLDR - You don't know what you are talking about.
- Joined
- May 1, 2020
- Messages
- 5
Nah mate, Gelbooru/Danbooru have comparable or bigger traffic (i'm not talking about bandwidth) to this site. Compare https://i.imgur.com/piCifze.png (your own analytics) to Danbooru's. The only advantage Danbooru (but not Gelbooru) has is that it serves pages through Cloudflare, which drives down bandwidth usage. Gelbooru's cost is probably higher than Danbooru, also contributing to the fact that they run older unoptimized code (Danbooru1), I'm not sure if they still colocate but I doubt they're paying $750 for that kind of server alone.
Also, the danbooru DB server is on nvme drives too. NVME servers at the lowest level still go for $50, a server for this kind of traffic is nowhere near the ballpark of $250/month, unless you're being ripped off.
Also, the danbooru DB server is on nvme drives too. NVME servers at the lowest level still go for $50, a server for this kind of traffic is nowhere near the ballpark of $250/month, unless you're being ripped off.
- Joined
- Feb 16, 2020
- Messages
- 462
@nonamethanks jfyi, MD @ Home is pushing about 10PB per month. A cool ~27 times more than Danbooru
mangadex.network is a good place to look at this ; consider this is only images, and only ~35% of traffic is reported, due to apps not being compliant with reporting for example
mangadex.network is a good place to look at this ; consider this is only images, and only ~35% of traffic is reported, due to apps not being compliant with reporting for example
- Joined
- May 29, 2012
- Messages
- 594
@nonamethanks
Okay.
Compared.
So they do 377TB in a month.
We do a several PB a month.
Not even on the same scale. We dwarf their traffic. Which makes total sense when you consider how many images a chapter contains, how many chapters are hit every minute, versus people opening a few images at a time there. And that's STILL excluding all the database access, api hits, webserver hits, etc that require the high spec servers to handle. You seem to be confusing storage space/bandwidth with actual server usage and just how taxing on servers it is to process thousands of requests and database updates per second.
TLDR - You don't know what you are talking about.
Okay.
Compared.
So they do 377TB in a month.
We do a several PB a month.
Not even on the same scale. We dwarf their traffic. Which makes total sense when you consider how many images a chapter contains, how many chapters are hit every minute, versus people opening a few images at a time there. And that's STILL excluding all the database access, api hits, webserver hits, etc that require the high spec servers to handle. You seem to be confusing storage space/bandwidth with actual server usage and just how taxing on servers it is to process thousands of requests and database updates per second.
TLDR - You don't know what you are talking about.
- Joined
- May 1, 2020
- Messages
- 5
Yeah, which you don't pay for since it's served by MD @ Home clients. Or do you do a database update on every single page read? I guess the absurd cost is driven by the fact that you use memcached instead of something better like redis.
An enterprise-level Hetzner NVME 6TB server is €133/mo btw. €165 with an additional 16TB HDD.
An enterprise-level Hetzner NVME 6TB server is €133/mo btw. €165 with an additional 16TB HDD.
- Status
Similar threads
