New servers and file optimisation request!
- Thread starter ixlone
- Start date
- Status
@Rugid um ... could you be kinder, my heart is a little disturbed by your sentence, if you want to give advice please use a better sentence, you know. And I just used Photoshop one year ago by self-taught learning, so I still don't understand. but thanks for your advice, maybe it's just me being stupid so I can't understand that ...😞
Power Uploader
- Joined
- Dec 16, 2019
- Messages
- 225
Wow, I thought PNGGauntlet'd be good but it took 3min for a compressed size 300KB bigger than Pngyu, both lossless. Guess Pngyu's (it uses pngquant) indeed the best when it's able to go lossless, unless PingaGUI is better than PNGGauntlet
Fed-Kun's army
- Joined
- Jan 25, 2018
- Messages
- 521
@Santiasan
Okay, I see what's going on. You are working with low-resolution, heavily-compressed JPGs of full-color images. The very first link in your post, which is supposed to be the original image (but clearly isn't because the text has already been removed from it), is already destroyed by compression artifacts. PNG will not help, and neither will reducing the color depth to 8 bit.
First off, where do you (or the cleaner who removes the text) get the original image from? I ripped out a page from chapter 1 hosted on Kakao which I assume to be the raw (720×1098, 66 KB), using the browser's built in Inspect Element tool. By itself, the image is already well-optimized (Kakao clearly cares about preserving bandwidth!), but since we still need to typeset it, we'll have to decide how to compress the result when we're done editing the image.
Doing so in Photoshop CS6, here's what I got by recompressing it with JPG (Save for Web... → JPEG, 80%, progressive) and running it through File Optimizer to trim off the fat: 91.2 KB. Yeah, we got a 38% size bloat, but we got it to look nearly indistinguishable from the raw. As the original was already well-compressed with no leeway for safe size reduction, if we had tried to match its size, we wouldn't be able to avoid noticeable quality loss. So that's how JPG fares.
How about PNG? Since the source is full-color and has soft color gradients, the only option to avoid major quality loss would be to save it as 24-bit, fully preserving fidelity at the cost of bloating the size to 302 KB (a massive 358% increase!). Worth it? I think not. Reducing the color depth to 8 bit would allow us to just barely beat the 80% quality JPG's size but immediately produce horrible color banding, so that is out of question. Even dithering won't help avoid very noticeable grain, not to mention it would explode the filesize again in this case (131 KB). Don't bother with it. Finally, there's the unpopular option of using the lossy mode of PNG which can be accessed by e.g. checking "allow lossy optimizations" on File Optimizer's PNG tab. This way we can reduce the size by a 1/3 to 203 KB at the cost of a minor fidelity loss (which in this case, somewhat ironically, smoothened out some of the artifacts originally present in the raw). Worth it? Not at all.
Hence, use JPG with high quality settings (80–85%, optimized or progressive) for full color digital images. And don't use second-hand originals; take the raws from the source if possible. Free webtoons allow easy (if a bit tedious) extraction of images from the online reader pages.
Okay, I see what's going on. You are working with low-resolution, heavily-compressed JPGs of full-color images. The very first link in your post, which is supposed to be the original image (but clearly isn't because the text has already been removed from it), is already destroyed by compression artifacts. PNG will not help, and neither will reducing the color depth to 8 bit.
First off, where do you (or the cleaner who removes the text) get the original image from? I ripped out a page from chapter 1 hosted on Kakao which I assume to be the raw (720×1098, 66 KB), using the browser's built in Inspect Element tool. By itself, the image is already well-optimized (Kakao clearly cares about preserving bandwidth!), but since we still need to typeset it, we'll have to decide how to compress the result when we're done editing the image.
Doing so in Photoshop CS6, here's what I got by recompressing it with JPG (Save for Web... → JPEG, 80%, progressive) and running it through File Optimizer to trim off the fat: 91.2 KB. Yeah, we got a 38% size bloat, but we got it to look nearly indistinguishable from the raw. As the original was already well-compressed with no leeway for safe size reduction, if we had tried to match its size, we wouldn't be able to avoid noticeable quality loss. So that's how JPG fares.
How about PNG? Since the source is full-color and has soft color gradients, the only option to avoid major quality loss would be to save it as 24-bit, fully preserving fidelity at the cost of bloating the size to 302 KB (a massive 358% increase!). Worth it? I think not. Reducing the color depth to 8 bit would allow us to just barely beat the 80% quality JPG's size but immediately produce horrible color banding, so that is out of question. Even dithering won't help avoid very noticeable grain, not to mention it would explode the filesize again in this case (131 KB). Don't bother with it. Finally, there's the unpopular option of using the lossy mode of PNG which can be accessed by e.g. checking "allow lossy optimizations" on File Optimizer's PNG tab. This way we can reduce the size by a 1/3 to 203 KB at the cost of a minor fidelity loss (which in this case, somewhat ironically, smoothened out some of the artifacts originally present in the raw). Worth it? Not at all.
Hence, use JPG with high quality settings (80–85%, optimized or progressive) for full color digital images. And don't use second-hand originals; take the raws from the source if possible. Free webtoons allow easy (if a bit tedious) extraction of images from the online reader pages.
- Joined
- Mar 29, 2019
- Messages
- 27
This shouldn't be optional, it should be an automatic part of what MangaDex does when images are uploaded - lossless file size optimization. Wouldn't hurt to carefully go apply that logic to all current images too.
If it's not automatic, than uploads should FAIL if the file size is > 5% more than the optimized version.
This policy is to not allow people to carelessly harm the bandwidth of everyone else. Even before the bandwidth problems some images were clearly 10x larger than they needed to be, and some chapters would take a couple minutes to load.
If it's not automatic, than uploads should FAIL if the file size is > 5% more than the optimized version.
This policy is to not allow people to carelessly harm the bandwidth of everyone else. Even before the bandwidth problems some images were clearly 10x larger than they needed to be, and some chapters would take a couple minutes to load.
Fed-Kun's army
- Joined
- Jan 25, 2018
- Messages
- 521
@SirDuncan
I believe the reason why this is a call to uploaders and not a mandatory server-side procedure is that the server-side resources are limited and could create upload congestion on busy days if every chapter is processed this way upon uploading. Besides, it's better if uploaders pay more attention to this, too, and look into other aspects of size optimization on their own (such as the ones I posted earlier).
It's impossible to tell how far off the mark the uploaded files are unless you attempt to optimize them upon retrieval. So both of your suggested options presume server-side optimization.
I believe the reason why this is a call to uploaders and not a mandatory server-side procedure is that the server-side resources are limited and could create upload congestion on busy days if every chapter is processed this way upon uploading. Besides, it's better if uploaders pay more attention to this, too, and look into other aspects of size optimization on their own (such as the ones I posted earlier).
- Joined
- Mar 29, 2019
- Messages
- 27
@moozooh
Yes, it needs to be detected eventually either way. I'd favor it being automatic, it should just happen like every other image posting service on the internet.
If posters want control of the minimization process settings, they should take change of the minimization ahead of time. The 5% (or 10%) allows MD to see that the user already did the minimization and to let the file pass unmodified.
Then uploadee sees either "Files are minimized, thank you, and will not be modified" OR "Your files are not minimized and have been automatically shrunk by 43%". Everyone wins.
But yes I realizes compute and programming resources are in short supply.
Yes, it needs to be detected eventually either way. I'd favor it being automatic, it should just happen like every other image posting service on the internet.
If posters want control of the minimization process settings, they should take change of the minimization ahead of time. The 5% (or 10%) allows MD to see that the user already did the minimization and to let the file pass unmodified.
Then uploadee sees either "Files are minimized, thank you, and will not be modified" OR "Your files are not minimized and have been automatically shrunk by 43%". Everyone wins.
But yes I realizes compute and programming resources are in short supply.
https://www.similarweb.com/website/mangadex.org#overview says about 40M page views per month = ~1.3M per day which is easily doable on micro AWS servers.
If using WordPress, caching can be a huge benefit!
Also what about a free tier CDN? Then could help alleviate bandwidth costs, though hitting limits is probably likely.
If using WordPress, caching can be a huge benefit!
Also what about a free tier CDN? Then could help alleviate bandwidth costs, though hitting limits is probably likely.
Fed-Kun's army
- Joined
- Jan 25, 2018
- Messages
- 521
@SirDuncan
I mean if MD optimizes on their end, there is actually no reason to check if uploaders do that. It won't decrease the amount of work in any way.
I mean if MD optimizes on their end, there is actually no reason to check if uploaders do that. It won't decrease the amount of work in any way.
@Santiasan He doesn't mean any harm. I work with people like that, blunt to a point but not malicious about it. Don't let it get to you, dude's helping people and helping the community regardless of how it comes across.
- Joined
- Oct 23, 2018
- Messages
- 26
I think I see some scanners in here, so I just wanted to thank everyone for their work. As someone who has none of the technical skills to do any of this, it means a lot that you all put your all into getting all the manga you do translated for us.
- Status
