Add last upload date for each language, to make the sort more accurate

Joined
Jan 18, 2018
Messages
45
Add a series of date fields to the Manga table to record when the last chapter of that language was uploaded.
EN_LAST_UPLOAD, FR_LAST_UPLOAD, and so on.


This would allow the 'sort by last uploaded' date to have meaning, as it currently simply pulls the last upload date of any language, often leading to users expecting an update and seeing no new chapters.

This would also allow cleaning the list based on if any chapters exist for the user's preferred language at all with minimal additional search overhead.
 
Upvote 2
Contributor
Joined
Jan 19, 2018
Messages
420
Short answer... No.

Longer answer... No because that would be extremely wasteful as there are lots of languages that the site supports but most titles only have a chapters translated in a couple of languages.

There are other ways to potentially get what you want, but they come with trade offs, usually performance related, that just don't justify doing it.
 
Joined
Jan 18, 2018
Messages
45
there are lots of languages that the site supports but most titles only have a chapters translated in a couple of languages.
That's exactly why this would be a useful feature. There are plenty of times where I search for new manga, open everything on the first page, and none of them have English chapters.
That's 10 page loads+queries that just didn't need to happen.

and I outlined a database structure that wouldn't cause much additional overhead.
-add update date for languages to the 'Manga' table
---Potentially this should be in a blob of some kind, but frankly there isn't a lot of overhead to adding a hundred mostly empty datetimes, beyond the normal timesink of adding new columns to an existing table.
-on uploading a manga chapter, update this date for that language
-on search use this date instead of however the query currently pulls most recent. at worst this is ~equal time, but if the current query joins to the chapter list to get last upload this would be way faster.

Heck, even if it is just a true/false 'Has chapters in X language' bit field it would help reduce pointless page loads for empty mangas.
 
Contributor
Joined
Jan 19, 2018
Messages
420
and I outlined a database structure that wouldn't cause much additional overhead
...sure...

Heck, even if it is just a true/false 'Has chapters in X language' bit field it would help reduce pointless page loads for empty mangas.
This already exists in the filter settings of advance search
 
Dex-chan lover
Joined
Jun 5, 2018
Messages
2,090
It doesn't need an additional field, since it's a very easy database query, but it should definitely be the default behavior for the search page and should be implemented for the "latest update" page too. And the "has chapter in lenguage" is only part of the solution, I am looking for manga that had recent updates in a lenguage I can read
 
Contributor
Joined
Jan 19, 2018
Messages
420
It doesn't need an additional field, since it's a very easy database query, but it should definitely be the default behavior for the search page and should be implemented for the "latest update" page too. And the "has chapter in lenguage" is only part of the solution, I am looking for manga that had recent updates in a lenguage I can read

There are plenty other threads that go over the complexities of MD situation and why "its just a simple database query" rarely applies. Needless to say, but if there was really just a "simple" solution that had little to no tradeoffs it would have been implemented long ago.
 
Joined
Mar 23, 2023
Messages
17
There are plenty other threads that go over the complexities of MD situation and why "its just a simple database query" rarely applies. Needless to say, but if there was really just a "simple" solution that had little to no tradeoffs it would have been implemented long ago.
What about making a lookup table instead, which would allow no unnecessary empty date fields. Something like "language_last_updated" which would be a table associated with each manga/comic, and have a "manga_id", "landuage", and "last_updated". This way you can avoid adding language date entries to a manga for each language.
I do believe this is a necessary feature, and it really affects the discovery experience in my opinion.
 
Yuri Enjoyer
Staff
Developer
Joined
Feb 16, 2020
Messages
425
I'm always a bit puzzled that people believe we're too dumb to think of trivial SQL solutions... The search is made using Elasticsearch in the first place, so there's no SQL available nor involved.

And yes we can (and maybe eventually will) order searches by last uploaded by language but you have to remember that most people have either no language filter (in which case this does nothing) or 1 or more. So there's a UI problem there to force the user to pick a single language to use for that ordering (or worse, multiple, in which case how do you handle conflicts per-language).

Either way, it will happen eventually, but it's not that trivial, as outlined in the other such suggestion threads
 
Dex-chan lover
Joined
Jun 5, 2018
Messages
2,090
I'm always a bit puzzled that people believe we're too dumb to think of trivial SQL solutions... The search is made using Elasticsearch in the first place, so there's no SQL available nor involved.

And yes we can (and maybe eventually will) order searches by last uploaded by language but you have to remember that most people have either no language filter (in which case this does nothing) or 1 or more. So there's a UI problem there to force the user to pick a single language to use for that ordering (or worse, multiple, in which case how do you handle conflicts per-language).

Either way, it will happen eventually, but it's not that trivial, as outlined in the other such suggestion threads
I never thought for a second that you were dumb and what you people did with Mangadex is nothing short of amazing.
That said, the fourth time you answer "we can't because we use elasticsearch" when people ask for properly working and properly sorted searches, one start wonder wether that was the proper tool for the job or if it was even meant for this kind of job in the first place. Also, from what I understood the database is already sorted by date, you just have to drop the results in a lenguage not in my list.

What REALLY baffles me, is that you ALREADY implemented it for the "latest updates" page, you just need to aggregate the results by title...
 
Last edited:
Contributor
Joined
Jan 19, 2018
Messages
420
I never thought for a second that you were dumb and what you people did with Mangadex is nothing short of amazing.
That said, the fourth time you answer "we can't because we use elasticsearch" when people ask for properly working and properly sorted searches, one start wonder wether that was the proper tool for the job or if it was even meant for this kind of job in the first place. Also, from what I understood the database is already sorted by date, you just have to drop the results in a lenguage not in my list.

What REALLY baffles me, is that you ALREADY implemented it for the "latest updates" page, you just need to aggregate the results by title...
Once again you seem to think that it super simple, and that for just arbitrary reasons it hasn't been done. There are reasons that it hasn't been done and it mostly comes down to performance and how elasticsearch works. For sure there might be "better" tools out there, but you have to ask are they better for Mangadex.

Consider reading the dev blog so that you can get a better understanding of all the limitations that MD has to deal with, and why certain tools and technologies were chosen.

Edit: To be clear no one is saying that it cannot be done, its just not worth the effort at this time to figure out how to do it without causing the site to implode on itself.
 
Last edited:
Yuri Enjoyer
Staff
Developer
Joined
Feb 16, 2020
Messages
425
when people ask for properly working and properly sorted searches
The search is working, and properly sorted. Your problem is that the sorting criteria you want (sorting by last upload in a specific language [or set of languages]) does not currently exist.

one start wonder wether that was the proper tool for the job or if it was even meant for this kind of job in the first place
One might wonder; however there's a reason why some 9 out of 10 websites implementing a powerful search use Elasticsearch for it. But maybe you know better than us and the entire rest of the industry despite clearly not having worked on the same issue before.

And using SQL searches was also the reason why the search was not even enabled for guests in v3. It is easier to make complicated searches with it, but you will never pull a fast multi-criteria search like ours (titles, tags, exclusions, various sorts, ...) with raw SQL searches, or you'll have other tradeoffs (needing much more compute, needing indexes on nearly all rows causing terrible writes and massive storage usage, ...).

This is specifically why no one actually does that and instead uses Lucene, Elasticsearch, and others.

Also, from what I understood the database is already sorted by date, you just have to drop the results in a lenguage not in my list
Again, it is entirely irrelevant how the layout of the data is in our database (and there's also no inherent sorting in the database, and the database also doesn't store every manga's last-uploaded-chapter-date-per-language because that would be ludicrous in its own right anyway.

At this point, I can't say if:
  • you believe we lie and "hide" a feature that is trivial just for fun, or
  • fundamentally misunderstand how it goes together

If it's the former, why? Why would we not if it was that easy and "already there" like you seem to believe? At least come up with a reason why we wouldn't want to offer it, idk. And if it's the latter, you should read https://mangadex.dev/an-api-to-rule-them-all/#data-caching, which mentions the general way searches are performed.

What REALLY baffles me, is that you ALREADY implemented it for the "latest updates" page, you just need to aggregate the results by title...
It is not at all the same. The latest updates page is a chapter search. And the reason Elasticsearch allows us to deliver near-instantaneous searches, have it enabled for guests, and so on is specifically because it's a document database. And document databases (most so-called NoSQL databases) make some tradeoffs like generally not supporting JOINs amongst other things. So there's no "just need to aggregate by title". It's literally not a thing nor how to do it.

Instead what we need to do is store, against each manga document (ie in ES, computed at upsert time of the document), a searchable field per-language for the last uploaded chapter data. It is feasible, and as we said many times:
it will happen eventually

Be patient, or at least keep your complaints to yourself if they're going to incoherently boil down to "but you said it 4 times already, but it's trivial and you did it already, and you could do it like that easily"...

We said it will eventually happen, and it will. No we cannot do it how you think we could, or we would already have done so.

If your complaint is purely that it's not yet done: this falls into the (very) long list of things that are doable, but a bit tricky, but we want them and many users also do. That list is filled with good ideas, but days are made of only 24 hours and we all have dayjobs.
 

Users who are viewing this thread

Top