Add/redesign volume entity

[boursk]

Problem:
As new manga chapters get released, they can't be assigned to a volume, since they can only get published after a certain amount of chapters have been created and published in magazines. This leads to the inherent problem of having to bulk edit chapters to assign them to volumes after they're published. Apart from that, a lot of uploaders make mistakes or simply can't be bothered to look up the volume numbers of the chapters they upload.

Solution:
Create a volume entity that has the start and end chapter (note: end chapter 25, should also include 25.x) defined and remove manual input of volume numbers. The volume entity could also have links to the volume cover images (if the art MangaDex only consists of volume covers) and I also saw a suggestion mentioning volume titles which (if relevant) could also be added under this entity.

The idea is to add volumes to MangaDex as they get published. Once set, the chapters can then be grouped by volumes on-the-fly whenever the chapter list is shown on the page. This doesn't take away all maintenance regarding volumes, but it massively reduces it. There's no need to care about mistakes too much when doing potentially expensive and error-prone bulk edits, because adding a volume is just a single insert and mistakes are easily fixed with a single update-statement. And there's no need to care about wrongly individually entered volume numbers due to mistakes or laziness, because the system will know the correct volume if/when it is added.

As far as the implementation goes, there's no need to link the chapters to the volumes. Just have both chapters and volumes linked to the manga. Volumes shouldn't need to infer the manga chapters it holds based on chapter entries. The volume record should have fields for the start/end chapter which can have numbers beyond what is currently available on MangaDex. That allows for newly inserted chapters to automatically be shown under the correct volume. It also avoids potential single source of truth issues and removes the need to create a default volume to link the chapters without a volume.

 

[Valle_reading_manga]

Related threads https://forums.mangadex.org/threads/add-volume-info-from-edit-title-page.1537679/ and https://forums.mangadex.org/threads/move-volume-metadata-to-manga-a-separate-entity.1916037/

 

[Valle_reading_manga]

As far as the implementation goes, there's no need to link the chapters to the volumes. Just have both chapters and volumes linked to the manga. Volumes shouldn't need to infer the manga chapters it holds based on chapter entries. The volume record should have fields for the start/end chapter which can have numbers beyond what is currently available on MangaDex.

While this sounds nice on paper, it's not quite as straightforward, since not all titles have chapters with ascending chapter numbers for various reasons. There's a manga API attribute called chapterNumbersResetOnNewVolume which was created to kinda just solve that problem, but I honestly can't remember the details nor has there been any progress on that front in forever afaik...

 

[boursk]

Related threads https://forums.mangadex.org/threads/add-volume-info-from-edit-title-page.1537679/ and https://forums.mangadex.org/threads/move-volume-metadata-to-manga-a-separate-entity.1916037/

That's wild. I tried searching for thread names in the (main site?) suggestions forum containing the word volume and I couldn't find anything relevant at the time. I must've done something wrong there... 😓

While this sounds nice on paper, it's not quite as straightforward, since not all titles have chapters with ascending chapter numbers for various reasons. There's a manga API attribute called chapterNumbersResetOnNewVolume which was created to kinda just solve that problem, but I honestly can't remember the details nor has there been any progress on that front in forever afaik...

I'm going to need some kind of example that illustrates what you're talking about, because I don't see why it would be an issue to say:

volume_number	chapter_start	chapter_end
1	1	5
2	6	11
3	12	17

And just group chapters with chapter_number >= chapter_start && chapter_number < (chapter_end + 1)
It doesn't matter if there are gaps or if there are decimals. Unless you're saying (in the above example) chapter 15 could also be in volume 1. I really wouldn't know why or how that would make any sense, but then the typical way to solve that, is to have a separate field for the internal value and display value. The display value can be whatever the official chapter number should be and what the user should see in the frontend and the internal value can be whatever makes the most sense for the internal systems to work with.

If you can find a real-life example of what you meant, I would be grateful if you could share it.

 

[Valle_reading_manga]

I'm going to need some kind of example that illustrates what you're talking about, because I don't see why it would be an issue to say:

volume_number	chapter_start	chapter_end
1	1	5
2	6	11
3	12	17

And just group chapters with chapter_number >= chapter_start && chapter_number < (chapter_end + 1)
It doesn't matter if there are gaps or if there are decimals. Unless you're saying (in the above example) chapter 15 could also be in volume 1. I really wouldn't know why or how that would make any sense

Well, chapter numbers can not be unique per title (hell, not even per volume probably, seeing how some authors seemingly just arbitrarily decide on chapter numbers) for various reasons:
MD uses volume numbers to designate Manwha/Manhua seasons and many titles just start their chapter numbers over for each volume/arc/change of magazine/you name it.
And then there's just semi arbitrarily numbered stuff or the results you get when extras from e.g. a pre-serilalisation are included in the serialisation's volumes, and the list goes on...
Therefore, the attribute I mentioned earlier was intended to toggle between the current behaviour for the edge cases and some mechanism similar to what you suggested iirc

but then the typical way to solve that, is to have a separate field for the internal value and display value. The display value can be whatever the official chapter number should be and what the user should see in the frontend and the internal value can be whatever makes the most sense for the internal systems to work with.

Unfortunately, display and internal value are the same thing on md. While I agree a display value would be needed, it comes with all the downsides you listed and would have to be added/edited retroactively across a huge number of chapters...

---

As for examples, I'm kinda too lazy (and on the phone) atm, so I'll get back to this another time, hoping someone else takes care of it in the meantime...

 

[boursk]

As for examples, I'm kinda too lazy (and on the phone) atm, so I'll get back to this another time, hoping someone else takes care of it in the meantime...

You could've just waited until you had the will, time and ability to write the post with examples. It makes the discussion a lot easier if we both know what you're talking about. 😅

MD uses volume numbers to designate Manwha/Manhua seasons and many titles just start their chapter numbers over for each volume/arc/change of magazine/you name it.

That's an interesting case. It also sounds like that was what the switch you were talking about (chapterNumbersResetOnNewVolume) is for. Like I said, I would just add an internal value that would work well with the volume determination and you'd just have the official chapter number as the display value. I can't say how I would design the internal value for this use case, because I don't have enough information on how these chapters and volumes/seasons are released. But I'm very confident that this can be figured out and that this value for this particular use case can be generated as long as the rules are determined.

And then there's just semi arbitrarily numbered stuff or the results you get when extras from e.g. a pre-serilalisation are included in the serialisation's volumes, and the list goes on...

Afaik, this is solved by using decimals. Therefore I don't see the issue. If you have an example where this is particularly weird and would lead to a strange mapping with the volumes, then I'm eager to see it.

Unfortunately, display and internal value are the same thing on md.

Of course it is right now. That's why I'm saying that if there are particularly weird things going on with the chapter number/volume number mapping (like the manwa/manhua example you mentioned), an extra field would need to be added next to the already existing chapter number. The new field would for the most part hold the exact same value as the existing field, but (presumably for the manga entries with the chapterNumbersResetOnNewVolume = true value) it would have to be generated where necessary.

Anyway, I don't mind going into the weeds, but the point is that all of this is perfectly solvable. If your argument is just that devs need to think carefully about this feature when implementing it, then great. Let them figure it out. And (just to not sound too irresponsible) in the distant off-chance that they would want help with it, I don't mind helping out.
If on the other hand you're convinced that the suggestion is fundamentally impossible due to certain edge cases that you're aware of, then we can definitely discuss this further once you've found the will and ability to look up those examples.

 

[Valle_reading_manga]

but the point is that all of this is perfectly solvable.

I don't disagree with that, all I said was

While this sounds nice on paper, it's not quite as straightforward

There's quite a bunch of edge cases (which I've mentioned some of), making your suggestion quite a bit more complex.
Which in turn increases the work required to implement it, so I wouldn't get my hopes up...

 

[boursk]

There's quite a bunch of edge cases (which I've mentioned some of), making your suggestion quite a bit more complex.
Which in turn increases the work required to implement it, so I wouldn't get my hopes up...

Maybe I'm underestimating the number of different types of edge cases (I've only seen one concrete edge case you've mentioned) and the volume of it, maybe you're overestimating just how much of an impact these edge cases have when it comes to implementing this feature.

The reason why I suggested it in the first place, was because it would greatly reduce the amount of maintenance needed and adds additional fool proofing for the overwhelming majority of the database. Even in the off chance that some edge cases need special attention, it's worth implementing it for all the manual work it saves the mod team and the users who submit reports. Maybe you disagree, but I'd say those are benefits worth putting in the effort for.

Software engineering is all about working hard in the present in order to be lazy in the future.

 

[erova]

Mm this is more complex than it looks and has been requested quite a bit over time, i seriously doubt it'll happen. i'm currently working on a project trying to cover this, so to pull it off you'd need a major refactor in how chapters work, since there are chapters or tocs that can get pretty cursed.

numbers resetting per volume like already mentioned, chapters out of order, arc-based numbering, volumes without tocs, numbering that doesn't match the serialization, chapters not compiled into volumes, etc. at least if you're accounting for all the possible edge cases. but for the majority, it works fine.

the way i see it, you'd need a sequential numeric index that serves as your chapter range for the volume entity, and another index for the cases i mentioned. let's use Mystery to Iu Nakare as an example.

Index (Numeric)	Index (Alpha Numeric)	Chapter Title	Vol
1	1		1
2	2		1
3	Extra Chapter		1
4	2		2
5	3		2
6	4		2
7	Extra Chapter Episode 2 & 3		2
8	Extra Chapter		2
9	4-2		3
10	4-3		3
11	4-4		3
12	4-5		3
13	Extra Chapter		3
14	4-5		4
15	5		4
16	6		4
17	7		4
18	Extra Chapter		4
19	8		5
20	8-2		5
21	8-3		5
22	8-4		5
23	Extra Chapter		5
24	9		6
25	2.5		6
26	2.5-2		6
27	2.5-3		6
28	Extra Chapter		6

The chapters in Mystery here are used as cases or arcs. each number represents a specific case or arc, since the series is about a detective. So the numbering might not make sense at first without that context.. so cover edge cases, it just makes things way more complex.

i'm not saying it can't be done, this is just an example. there are also well‑known cases like One Punch Man, which has been a nightmare, with chapter numbering differing between the Tonari no Young serialization and the volume releases, basically merging two or more chapters.
to use a practical example, the latest OPM chapter as of today is 271, but the last chapter of the latest volume is 188. since there's no way there's almost a 100‑chapter gap, you can kind of guess what's going on, right?
well, chapter 188 of volume 36 is actually chapter 228 of Tonari no Young.

so in that example, you'd have to reuse the alphanumeric index numbering. and while the simplest approach is to treat that volume entity as starting at chapter 1 and ending at chapter 5 (like you mentioned) or just use vol 1: 1-5 chapter, but you'd still need to use the numeric index and normalize the numbers, For that, you'd need to either normalize the chapters, follow the ToC numbering, or ignore it or have both.

but i can agree that for most cases it could work or speed things up a bit and avoid manual or bulk editing of individual chapters.

 

[RogueKitsune]

May I direct your attention to this bundle of nightmares joy
https://forums.mangadex.org/threads/weird-edge-cases.1195912/

 

[boursk]

Oh man, I love this! I'm getting more excited looking at this than I probably should. 😅

the way i see it, you'd need a sequential numeric index that serves as your chapter range for the volume entity, and another index for the cases i mentioned.

I agree, that would be the obvious solution. That's what I meant with an internal value (the former case) and a display value (the latter case) for the chapter number.

As far as the cases like Mystery to Iu Nakare and One Punch Man; these are situations that have been solved in the past. So they can easily be converted in the new system. Mystery to Iu Nakare apparently just uses its own numbering and One Punch Man does the same thing like so many other manga entries: use decimals in their chapter numberings. If you use decimal numbers, you don't need to do anything special in regards to the proposed suggestion. Otherwise, choose one numbering system and stick with it. There's no need to maintain two separate numbering systems at the same time.

I understand that these are just illustrative examples of the kind of weird stuff that happens in the world, but what I'm really looking for are entries that are currently on MangaDex that make implementing this feature difficult (or just plain impossible). At the very least, these two entries don't need special processing to make them work in this new volume system.

But let's say Mystery to Iu Nakare was actually numbered the way you described. I could still write a small algorithm to convert these chapter numbers and volume numbers into a useable internal value. The same algorithm could be used for all entries that reset the chapter numbers with each volume. That would greatly reduce the perceived manual work needed to migrate to this system.

Things get very funky with examples like Gochuumon wa Usagi desu ka? (thanks to the thread @RogueKitsune provided for the example). From what I understand, the official numbering resets in the early volumes, but after volume 8, they just pretend like everything has been numbered across volumes. If everything was numbered in that way, it could be dealt with, but the uploaders of the Portuguese translated chapters has been using sequential numbering throughout the volumes that didn't have that. It's very fixable, but that definitely requires some manual intervention to make it work.

On a more disturbing note (again, thanks to RogueKitsune's link); Tonikaku Kawaii has a volume 16 that randomly has a chapter 1-4, then moves to chapter 148 and then has chapters 138-140 (plus an omake for 140.5).

Spoiler: Manga ToC

This has been properly numbered in MangaDex according to the official numbering. This would obviously also be an issue. You could add an internal value of the reading order to force it into the mold, but I think that in this particular case it would just be too confusing and near impossible to maintain. I don't know how many of these type of cases exist, but assuming they're very very small, I'd probably choose to have a volume number override value in the chapter record (basically the current volume field). The entry wouldn't enjoy the advantages of this suggestion, but that's fine. (If you leave the volume field in the chapter record as a backup for these highly irregular situations, it would completely eliminate all migration headaches for anything that doesn't directly work with the automated migration process. Those entries would still look and work according to the current system and could be fixed manually at a later date.)

Sometimes when you design a system, you might find that there are real world examples that don't quite fit the mold. Sometimes you can be smart about it and figure a nice way to make most of the irregular use cases fit the mold anyway, but sometimes you'll also have to be content that a handful of use cases are best left outside of the mold. That might sound like a cop out, but it can be an effective strategy to not be bound too much by the miniscule minority of exceptions and reject changes that would improve the system for the overwhelming majority.

It's just a matter of getting an inventory of how many issues there are and making a decision on whether it's worth it, based on the ratio and scale of the irregular entries and whatever complexities would need to introduced to deal with the entries don't fit in the mold. Given everything that has been discussed in this thread, I can still see a way to implement this in a way that is clean and maintainable. I hope that the MangaDex devs share this vision.

 

[erova]

I mean, the main issue isn't really the numbering, is how to reduce workload, you can set that value easily enough, but what comes after is dealing with chapters uploaded by scan groups. Some might use the magazine serialization numbering, others go by the volume release (which is exactly why this whole thing is such a pain to manage). You never fully escape manually editing chapters.

Take Gochuumon: the problem with chapters resetting per volume is that if you select a range like 1–13, that same set of numbers repeats across 7 volumes. You'd have to manually go through each chapter and match them based on the chapter title.

Tonikaku Kawaii is more of the same. If you have repeated chapter numbers like "1" across different volumes, you can't auto‑assign them, unless once a volume have assigned chapter, that chapter gets locked and only applies to the next available chapters. (assuming we're not dealing with a completed series and chapters are still coming out every now and then)

aaaaand sure, all of this can be handled manually with decimals o whatever, like keeping volume‑based numbering and just assigning the right chapter to the right volume, but your suggestion was about cutting down that manual, per‑chapter work. Unless the chapter indexing model changes, you can't really do it cleanly.

And while ignoring edge cases might be a valid approach, you're gonna run into a ton of headaches when you actually hit them. Which is kinda what led us to this convo in the first place.

 

[boursk]

And while ignoring edge cases might be a valid approach, you're gonna run into a ton of headaches when you actually hit them. Which is kinda what led us to this convo in the first place.

The point I was trying to make (and I understand if it might be buried in such a long post 😅) is not to simply ignore the edge cases that really can't fit the mold, but to have a backup system in place so that they can be safely ignored.

Here's a way how it could work:

You can have a system where you work with two indicators: has the manga entry been marked chapterNumbersResetOnNewVolume and/or are there no volume entries related to this manga entry? If so, show the volume field in the chapter insert/update screen. The process will work exactly the same way it does now. In case of the chapterNumbersResetOnNewVolume=true you could opt to add that extra internal chapter number value we talked about, or it can be discarded altogether.

For all the manga entries that do have volume entries, you can have a checkbox that enables the volume field. This would allow uploaders to override the volume number if there's weird stuff going on. This allows for the needed flexibility whenever the new system does apply to the manga entry.

When showing the chapters (this would be in runtime), you would pre-process the chapters by setting a volume for each entry by first checking if there's a volume number override value in the chapter and if not, try to find it within the volume entries. After that you can do the usual grouping by volume and ordering within volume before showing the chapter entries.

All this isn't overly complex nor hard to maintain. It provides the flexibility we already have, while still improving the QoL for the overwhelming majority of the entries. And on top of that, the migration to the new system can also be very painless, since you can decide just how much of the current chapters' volume numbers you want to transfer to the volume number override field.

I can understand if you're skeptical about the entire solution if there's no plan at all for the edge cases. So I hope this has eased that feeling a bit by showing an example of how it could be implemented. And hopefully you can agree with me that even if this new system doesn't really help cut down (nor increases) the maintenance for those handful of entries that really don't fit it, it's still a change worth doing for all the entries where it does work for.