New servers and file optimisation request!

[doppler]

AMD Ryzen processors are really cool.
The newest ones with 64 threads are kick ass. They run slower than today's fastest. With 64 threads who cares.
Pinga would find someway to bog them down too.

 

[orochijes]

Here's the best suggestion to save space you could possibly hear so take it to heart: delete chapters translated into languages almost nobody reads on this site. Hooray for exclusion!

Wait, you think I'm joking?

 

[Needhydra]

right now I cant tell if something is going wrong with compression aka changing grey values or my computer has decided to sample the images slightly differently due to my main manga just being mostly patterns for shading.

edit before anyone says anything its already 8 bit grey

 

[kardyro]

@doppler

limit cpu affinity and/or don't use the gui variant -- I've heard of people having issues with the gui. when I first found out about pingo, I threw it at my storage drives of images (after checking that it does-the-good-thing(tm)), limited to two cores. back then, it exploded on files ending with a

.

(e.g.,

titlepage..png

), so those had to be renamed, but otherwise it went without a hitch.

the ending numbers are also mostly accurate. if anything, it under-reports (by less than 0.01% though, so it's moot imo).

 

[Teasday]

_{@Aeder posted:}

Has Lossless WebP been considered as an alternative to PNG, now that all of the browsers that matter support it?

It was already mentioned but I'm obligated to once again moan about the fact that unfortunately, ityool 2020, iPhones are still things that people use.

 

[bk3k]

As for this subject, saving storage space is nice as is saving bandwidth for the servers. At no cost to quality, might as well. But you're forgetting other benefits. Your manga will be loaded faster. Especially nice for your poor mobile users. It takes saves RAM, again nice for those with older/cheaper hardware. Back to the part where there is no cost, no downside.

I ran the files through Pngout (Pnggauntlet) to lossless - no detail change - optimise them. The result is the total filesize dropped from 166mb to 77mb.

Results will vary radically. Some files I see > 90% improvements, some files something like 15%. Things with a huge amount of colors/detail won't shrink much while remaining lossless. But for the most part you are talking real life photography where it isn't worth it - as changing from RGBA to indexed isn't viable. For manga, game assets, etc it is very worth it. Also note PNGGauntlet is using PNGOUT, OptiPNG, and DelfOpt in series. So it will do better than just PNGOUT alone. But you could just as well use those tools in series via a script to optimize with no user input needed.

Pingo. Install PingaGUI, drag and drop files and click optimize. Pingo is much, much, much faster than Pnggauntlet.

But what gets better results? The time it takes PNGGauntlet is negligible if you are talking a single chapter of manga. Go cook a burrito. When I use it to optimize something more substantial a game's assets, I just let it go overnight and it gets done far before I wake up. Or when I'm heading to work, or I'll just be browsing the internet for a while and won't notice. Now if you are getting results just as good and if it is pixel-perfect, then I guess that's a step up.

As for the webp, I think you should pass on that. Not only excluding some browsers, but what of people who might save the images and want to view them in other programs? Don't use fringe formats.

But you people who use jpg... no. Just no.

 

[moozooh]

There's one little trick with PNGs that the savvy scanlators use to very good results and one that completely blows JPG out of the water in a fair comparison.

High-quality scans, especially when done on a high-detail manga with lots of shading and complex gradients, will always be very grainy: both because of the ubiquitous pattern fill and because the high resolution exposes printed ink granularity. This, by itself, is absolutely horrible for compression as PNG performs best on solid colors, and noisy/grainy images inevitably bloat its filesize. But it also works to your advantage if you know how to deal with it the best way, which is reducing the colors further and using dithering to compensate for gradient fidelity, which is something JPG cannot do. Since the image is already grainy due to the natural dithering of ink printing, it successfully masks any dithering artifacts that would have become more obvious on a manga with solid colors.

To give you an example just how well it actually works, let's take this beautiful two-page spread from Kokou no Hito. It is actually a JPG image, and is already pretty heavily compressed, but that doesn't matter too much as in this case it's basically impossible to tell with a naked eye where the artifacts are among all the visual noise (in fact, let's take that as a challenge to see how far we can push PNG is in this context).

Here's an Imgur album with all images fully optimized (including the original JPG), including diff comparisons. I used Photoshop CS6 to make PNGs and File Optimizer to crunch them out (notably, the best it managed to produce was 9% reduction on the most color-starved versions with 5–6% on average across the set—Photoshop is actually very good at this already). Filesizes are included in descriptions, and you can download the album and run your own diffs. Outtakes: at 32 colors PNG is completely indistinguishable from the JPG, at 24 colors dithered it almost catches up and would have overtaken the JPG by both size and quality if we had access to the original image, and at 16 colors dithered it still manages to compete in quality while having a comfortably lower filesize.

Here's another example with Vagabond: this time we have the original PNG so we can compare apples to apples. Results: 256 colors → 16 colors dithered, 991 KB → 600 KB flat. Can you spot the difference? Mind blown yet? This is the reason why e.g. all Berserk volumes released by Evil-Genius have at most 64 colors (and even then it's just contingency, you wouldn't have noticed it if they halved it). They just don't need any more. This will also be the case for any high-detail artwork.

 

[hollyguild]

I just wanted to say that I thought a lot of scanlation teams had forgotten how to make clean compressed files, just like how fansub/video ripper groups release everything as giant uncompressed monster files. I am glad to see there are still people who take the craft and my HDD space seriously out there. Thank you to everyone who puts effort into putting out the highest quality scanlations, some of us really appreciate it.

 

[AerialAtom]

@Dher I exclude these plugins on FileOptimizer cuz most of the time they take too much time and they don't cut enough to warrant using them
pngrewrite.exe;optipng.exe;PngOptimizer.exe;

And I run multiple instances with at least 5 images or leave it running overnight sometimes.

But yea, this program rocks.

 

[lnomsim]

I use Paint.net, not the best tool, I know, but the only one I can use without losing my hair.
I just tried to export my cleans into 8bit, it doesn't change the size of the files at all.

 

[Shin_968]

What about the Save for Web option in Photoshop with the following settings:

This is from p00kie's editing guide and is what I'm using right now. Pretty neat and convenient.

 

[Santiasan]

You know, I was just trying to match the one on the first page (export as png) and when I compared it to the original png and jpg which had maximum quality (in photoshop, 12). I found that the original JPG and PNG don't look different (smaller JPG size) but the exported PNG produces broken images and adds unnecessary color !! I will try another method to see the different quality~
edit: I think, I just will use JPG... with maximal quality, or maybe in PNG with medium save, that's better than export PNG, and about PNGGauntlet, I think that's really slow, I'm just trying 1 image about 1.5 Mb and until now it hasn't finished yet...

 

[QSS]

If you guys do implement JPG images, could you make it so that the default is original resolution? I want to force people to opt out of instead of opt in to my high res cleans, god dammit!

It's a real shame that optimization, color space, and how image formats work is not common knowledge among scanlators.

I really appreciate your comment, @Guspaz. Definitely learned something from that.

 

[night_]

Crawling the static storage once in a while is nice and all, but counting on all users to clean their stuff before uploading might be stretch.
If you can spare any resources, maybe baking an image cleaning/normalizing routine for new uploads somewhere in the backend may be a good idea.
ImageMagick and Pngquant are two FOSS utilities that could serve that purpose.

However on the off-chance any of you guys might feel like doing some hardcore devwork, there's this fancy algorithm you can try toying with.

 

[Khaos]

Likely won't save space but why not have both PNG and jpg and users can choose quality or speed

 

[erati]

Yes please all scanlators start doing this! I made a thread about this: https://mangadex.org/thread/242338
The thread also has a very basic pingo script that is a bit more consistent and doesn't slow your PC quite as much. You just drop the bat file in a folder to handle all the PNG's in folders and subfolders in a consistent way with no loss. Great for optimizing any manga you archived previously.

If you're a scanlator and don't want to change your software or routine that much you could just run this the script in the thread above on whatever png's you make and wait a bit to make it easier and still effective. Just do what you normally do, click the bat file and wait a bit.
EDIT: @doppler this script might be preferable to using the gui tool for some of the concerns you listed about pinga.

If anyone made their own scripts using pinga/pingo or any other software like it I would love to look at it.

 

[doppler]

@kardyro
The GUI does not assign cpu affinity when it spawns a thread. So it just blows through the CPU cores like butter. Not being a kick ass programmer myself I have been thinking and looking for generic GUI interface forms. Something kinda like shell of a GUI to handle files in and out. I then apply a core function to do (pingo) inside. And let it run. ANYBODY KNOW OF SUCH A PROGRAM ? Until then I will look at other things like:
@erati
Has posted his/her (script,bat). I will look at it today later. Erati, pinga is the best there is for PNG's. Nxpower lite desktop 8 is the best there is for JPG. Neither of them do the other file type good at all, in fact they do it badly. So uncheck JPG in pinga and delete the PNG dll file in Nxpower. Pinga won't do accidentally dropped JPG's and NXpower will complain it could not do the PNG.

There are two things I love about computers. 1. Compression of files and media. 2. Encryption/Code breaking. (all forms) There is a whole agency devoted to my second love. If more powerful computers were around for the Germans almost a century ago. Things would have way different today.

 

[Beregorn]

@Santiasan png is better for storing anything, since it is lossless (it saves the exact value of every single pixel) and has comparable size to the average jpg. Unless you don't care for the quality of the final result, then use jpg and compress the crap out of it.

 

[CatWEax]

@Shin_968
Those settings aren’t that good, to say the least.

17 indexed colors is stupid because that requires 5 bits. 16 colors requires 4 bits and compresses better. The reasoning for using 17 given in that guide is basically “it’s flawed, but that’s what we’ve always been using”. Well, don’t.

The explanation for disabling progressive JPEGs given in that guide is just plain wrong. Yes, a progressive JPEG is kinda like an interlaced PNG, but while enabling interlacing for PNGs generally increases file size (lines far apart are more different but are compressed together), enabling progressive encoding for JPEGs decreases file size by about 5-20% because it allows the encoded frequency data to be re-ordered to be compressed more efficiently. (Non-progressive is like “low mid high low mid high low mid high”. Progressive is like “low low low mid mid mid high high high” and compresses much better.) Progressive encoding of JPEGs should always be enabled unless you’re targeting ancient browsers (older than 15 years or so).

Quality 80 is kinda meh. Don’t go below 85 for anything but thumbnails.

@Santiasan
I doubt Photoshop would actually produce “broken images” when saving as PNG and how can it add color by itself? You must be doing something wrong. Also, make sure to change the image mode to grayscale before saving.

 

[Santiasan]

@CatWEax I think you missunderstood, I'm saving a manhwa so it can't be in grayscale mode:' and in the page color it become like a line,um...for example, I put a text with glow effect, and when I save it with "export as png + smaller file" the glow effect become a line, you can check it in your photoshop.. Btw, change the background to black so you can more easy to observe but thanks for your suggestion about storing manga, because I want to re-translate a few manga