New servers and file optimisation request!

[Santiasan]

@Beregorn are you talking about "large file size PNG" or " medium file size PNG" to storing the manga? And about JPG... i think I will rarely use it from now on, thanks for the tips!

 

[NodiX]

@Lymus

There's no reason not to use WEBP because some browsers don't support it. There are ways to serve WEBP images with fallbacks to either JPG or PNG. One of them is using <picture></picture> tags to wrap two different image types with WEBP as the main image and the other way is to check whether or not browser supports WEBP from the server, then serve best format appropriately. Some image processing library for transforming images on the fly such as Thumbor and Imaginary actually support parameter "auto" for this reason. I actually set up my own image proxy to load chapters from Mangadex API with Imaginary and Cloudflare Workers as the middleman to save me my internet data.

 

[CatWEax]

@Santiasan
So it doesn’t “add unneccesary color” but messes up the colors you added. I don’t have Photoshop to try, but the “smaller file” option supposedly reduces the number of colors in the image. How about you just don’t use the “smaller file” option if it breaks things?

Btw, if the color manhwa raws were already JPEGs and you didn’t remove the JPEG artifacts, you should save your edited images as JPEGs. Keeping JPEG artifacts and saving them as PNG is a serious waste of space. None of the tools suggested in this thread can make up for that.

NodiX said:

I actually set up my own image proxy to load chapters from Mangadex API with Imaginary and Cloudflare Workers as the middleman to save me my internet data.

Wow, and here I thought I was the only idiot setting up his own proxy with TLS-interception to cache all files from the image servers.

 

[me_no_kishi]

_{@Teasday posted:}

_{@Aeder posted:}
Has Lossless WebP been considered as an alternative to PNG, now that all of the browsers that matter support it?

It was already mentioned but I'm obligated to once again moan about the fact that unfortunately, ityool 2020, iPhones are still things that people use.

There are polyfills for Safari/iOS. 12% file size reduction is worth considering the tradeoff.

 

[NodiX]

@CatWEax

I don't actually use those to store images from Mangadex. Imaginary does optimize and resize images from Mangadex, but by default it's doing it on the fly. I'm using Cloudflare Workers to cache the result so it doesn't convert to WEBP twice if I get the same image later.

 

[kevinmul]

I mean... at this point shouldn't all the scanlators compressed their works already?
You know... saving internal storages, server's storages, upload time, bandwidth, etc...
I mean... you can open Patreon and ask the reader to be your patron to enjoy the lossless files hahaha

My business mind is scary... somtimes~


I am a noob about this compression file things...
If your server compress the file automaticly, wouldn't it be:
1. delay the time before reading?
2. worsen the already compressed files before upload?
3. taxing the server resources or need a new dedicated server?

Just pure curiousity ...

 

[Santiasan]

@CatWEax hm... so like that huh... by the way, "small file" make the size of the image become smaller but put some noise (the color that not needed) and if I did not use "small file" the quality is good but the size really big, you know..>.< for example I have an image that I save in 1.5 Mb (PNG without small file) the quality is 100% of 100% but when I save it in PNG with small file, the size becomes 756 bytes or kilobyte (I'm forgetting) but the quality really, ugh... not good (that what I get when I zoom in 400%, what about 1000%? I think if I zoom in 1000% it will become worse). So maybe I will just store it IN PNG with medium size, I hope the quality did not become worse>.<

 

[doppler]

Kinda funny how this discussion was month's ago when I had a thread going on this subject. An admin commented about "Mangadex" does not need to reduce and compress yet.

Then

Covid 19, mangarock (dead), the influx of 100's of refugee's from there. And the real big kicker in the mix. Scanlation groups with nothing better to do at home then do the thing they love.

Boom userbase exploded.
Boom Chapter's releases exploded.
Boom Mangadex usuage (low bandwidth days, distance memory) exploded.

Does anyone have PTSD or shell shock about this ?

 

[NodiX]

@kevinmul

1. No, converting image on the fly is actually one of web performance best practices that being encourage by performance experts (I learned about it from reading an article from a Google Engineer). Some image CDN services do this and cached images on CDN level, but for self-hosted version it requires some more set up to achieve balance between result and cost.
2. It depends on which compression settings are being used and how many times it being compressed. Generally websites that should worry about quality degradation from multiple compressions are websites with virality potential like Twitter and Imgur.
3. Self-hosted image processing required a new dedicated server, yes, but if being setup correctly it can save bandwidth cost especially if ~100MBs of images can result in a TB of bandwidth billing.

 

[moozooh]

@Santiasan
What's the original image you are working with (can you upload it somewhere?) and what is the software you're using to export and compress it (what particular programs, versions, etc.)?

 

[Lymus]

@NodiX to be able to serve both, without on the fly convert shenanigans, would mean we do have to store both. And it would mean an increase in cache space used up by different versions of the same image. Which is already causing problems.

 

[Santiasan]

@moozooh I'n working manhwa "who made me a princess" I upload the chapter not at the mangadex but in my blog, you can see it in my profil group. And the software that's I use is Photoshop CC 2019 (I think, because I start using photoshop one year ago)

 

[NodiX]

@Lymus

Is outsourcing to multiple cloud providers really not an option? You know, there are some more efficient solutions for performance problems besides upgrading server every time you hit the wall. Some of them can be implemented without some significant code changes to your backend and can be placed on top of your existing build.

I'm a frontend dev and been focusing web performance optimization since last year. I wouldn't call myself an expert but I know one thing or two to solve slow websites, especially since I've made similar website to Mangadex and in a separate side project made myself a way to optimize images from Mangadex for my personal use. I want to help with things I learned, really, but it's quite depressing for my suggestions to be dismissed without much thought ever since I posted my first comment here.

 

[Aeder]

@Holo

Ah, if half the user base is on iOS then that is indeed a massive blocker. I mean, you could still serve different images to different people but the savings wouldn't be across all of the userbase and the complexity goes up.

I guess the only hope left is to wait and see if JPEG XL gets implemented in ALL browsers and ALL editors this time around for a drop-in replacement.

 

[AB2000]

@ixlone In the end, is the problem only that some images are saved as 16-bit PNG when 8-bit PNG would be enough for most of them?

Should we also refrain from upscaling images with tools like Waifu2X? (since larger images will make files larger).

 

[Halo]

And it would mean an increase in cache space used up by different versions of the same image.

You could do it temporary. Make a reasonable server-side optimization for new chapters only and serve them for the first N hours/days. After that, remove it and host just the original. Most users won't notice the difference. I suppose this goes against the founding principles of MangaDex, but it's better than getting batoto'd.

 

[NodiX]

For anyone who thinks serve webp is complex, let me give this example.

This is from my image proxy I implemented with Imaginary and Cloudflare workers. It grabs images with Mangadex API, convert it to requested format, then cached at Cloudflare CDN.

WEBP version:
https://mangadex-tinify.prognovel.workers.dev/image?width=844&hash=991126fb5a193bb658c46733bb98ddcd&server=https://mangadex.org/data/&name=x3.png&chapter=894731

JPEG version, in case browser doesn't support it:
https://mangadex-tinify.prognovel.workers.dev/image?width=844&hash=991126fb5a193bb658c46733bb98ddcd&server=https://mangadex.org/data/&name=x3.png&chapter=894731

 

[BraveDude8]

@Aeder Safari doesn't support it, so anyone on iOS would be screwed. Plus, .webp is pretty bad at dealing with greyscale images.

@orochijes If "almost nobody reads" them, deleting them isn't going to make a difference. The issue is bandwidth, not storage space.

 

[ixlone]

@AB2000

We're only asking people to optimise their files before upload, not gimp their releases into something they wouldn't normally do.

 

[moozooh]

@Santiasan
No, see, I asked for the original image before you compress it with JPG, not the end result you post on the blog. If you had uploaded one of them somewhere we could take a look at it, we could perhaps give you some more specific pointers as to where things go wrong as you claim them to. That being said, if you don't post your work on MangaDex, it's probably irrelevant.