Making part of the site static

[Matrixik]

Just leaving idea because of problems with db when some big manga drop and get site to halt.

Maybe you could make chapter pages only as static html files that does not hit db on opening and are simply served with nginx (or what else you are using to serve this site). It could be used only for new chapters (like uploaded in last 24 hours) for specific manga (or all of them) and remove it for older chapters. You would need to recreate it on new chapter upload (for any group). Separate file for original and compressed images.

I can see only one things that need to be dynamic: if user read chapter, but this can be postponed to background with JS when user is reading chapter (so it can periodically ping API and stop if got 200/204 back).
You could recreate html with updated comment count on every new comment/no more frequently than 60 sec or something similar, it still will be lighter on server than connecting to db for every user to show them the proper count.

Why bringing the idea: when I moved my wordpress blog to static site (generated with Hugo) it became blazing fast. (for devs searching for some trendy name check JAMstack)

 

[EV-8371_kk]

I was thinking of something like this too.
I am not a web dev but everyone on the internet has an opinion.

Of course there are dynamic pages. People don't want to see certain mangas either by genre or scanlator groups. So those pages are generated for each user and then cached.

So my idea was just to static everything and let the user remove all the ones they don't want to see.

Any updates updates the static content and asks like users to ditch their local cache for the new one, then do the filtering themselves.

I think it is about time they start offloading some of this work to the client. Clients has it good for too long now.

 

[potatofood]

From this thread:

We currently serve static pages everywhere except the reader

 

[Teasday]

API hits for guests are pretty much cached at ddos-guard. The majority of the content we serve never even hits our servers.

 

[rdn]

hey @Matrixik
thats a good idea you have and something we definitely want for v5
you also identified the exact problem why we are having the database issues that we are currently struggling with.

first i'll give some context to what we're currently dealing with and then describe the plan we have for v5 and how we will deal with that problem in the future.

At the present time, our website and our api are not stateless. that means that everyone visiting sends a session id inside a cookie to the webserver and waits for a response.
The webserver then has to ask the database server for more details about this session. Are you a guest or a registered member? are you a member of one of the groups that uploaded this chapter? Or maybe a staff member?
The answer to that determines if you are allowed to access this chapter and which other options you have, like if you can edit or delete said chapter.
If you are a guest the answer is simple: Can anyone see this chapter? Then serve it. If the user is not a guest, we have to do a lot more checks based on data from the database first.

The first case is already cached by our ddos-provider, so what you propose is already happening. In fact, roughly 3/4 of our entire web traffic doesnt touch our infrastructure because they saw that the response is always the same. They only serve static html.

The second case is where it gets a bit more complex because we have to ask the database a lot more questions to figure out who you are to determine what we should show you.
Granted, the codebase is not in a good shape to handle this with the traffic we have, so these queries tend to pile up a lot. We have several database servers that only answer to questions (read-only slaves) but we can't predict which requests we get and the load varies a lot throughout the day.

v5 will have two major changes to deal with this issue:
the first is that our api will be stateless, which means we dont need to do any database lookups anymore to figure out who you are, because this information is inside what your session-cookie was before. I can explain this some more if you're curious.
The other major thing is that instead of building a huge query to the database to get the result, we use different types of databases that are better in storing and answering such questions compared to the mysql database we currently have. Namely elasticsearch and document-storage databases.

Hope that answers your question and gives a bit more insight.

 

[Matrixik]

@rdn thank you for long answer, but now I'm feeling that it will take a long time for v5 to arrive...