MangaDex Development Update - API Abuse Enforcement

[Tony_Mushah]

the User search function has been removed from the left hand menu on the mainsite

Actually, user search is already locked in the API since quite sometime. You need to have authentication to use it.

that's the thing, these types of bad actors tend to use a lot of IPs in order to get around rate limiting to bog down the servers over a longer period of time
by adding a hard limit, i'm assuming MD is trying to mitigate high, consistent server load

Only the mainsite got this hard-limit, the API is fine. (i hope forever)
I guess, the heavy traffic came from the main website which is actually over scrapped by bots and stuff.

Playing devils advocate here, but if you'd really want to stop API Abuse, why hasn't the actual API been limited yet?

Many community apps and tools depends on the API, it will be the final nail on the coffin if that were to happen.
I would crash-out if that were to happen.

Why would scrapers even need the frontend? As I said the API has no limits. You can just carefully code your scarper to not trigger rate limits there. Maybe rotate around the IP. It really isn't hard to make a functional scraper that doesn't have to touch the Fronted.

I think most scraping bots doesn't give a sh*t about the API, most of them are probably aggressive AI scraping bots that uses puppeteer to pick up data on multiple website at the same time.

Speaking of API though, I guess the only way to really enjoy reading title on MD from now on is to use community apps like Mihon or to built your own app from scratch.
I took the second route, it takes time to get things right but honestly it is a blessing in times like this.

 

[BestBoy]

Actually, user search is already locked in the API since quite sometime. You need to have authentication to use it.

User search appears to be back
@Angry_Panda

 

[EtherealNeko]

We have temporarily reverted this since too many guests were signing up, and the auth service was beginning to degrade. We will try again later.

 

[shinya78]

We have temporarily reverted this since too many guests were signing up, and the auth service was beginning to degrade. We will try again later.

Is this the part where I say... CALLED IT?

If they resort to flooding the database with accounts just to bypass the 10-chapter limit, well...

 

[Tony_Mushah]

We have temporarily reverted this since too many guests were signing up, and the auth service was beginning to degrade. We will try again later.

I hope i won't get banned for posting this.

 

[Mangamoz]

Is this the part where I say... CALLED IT?

Welp, that instantly backfired huh.

 

[b-no]

Why would scrapers even need the frontend? As I said the API has no limits. You can just carefully code your scarper to not trigger rate limits there. Maybe rotate around the IP. It really isn't hard to make a functional scraper that doesn't have to touch the Fronted.

This one's to combat scrapers that rely on headless browser scraping. Otherwise, you're just calling an API and it isn't technically scraping. lol

 

[EtherealNeko]

Client-side deployments allow quick testing of anything that would happen if you implemented the same things on the server-side. This is why it was done this way, rather than directly into the API. Rollbacks on the API aren't a walk in the park, especially not since we don't run a versioned API.

 

[panzerkampfwagyu]

Client-side deployments allow quick testing of anything that would happen if you implemented the same things on the server-side. This is why it was done this way, rather than directly into the API. Rollbacks on the API aren't a walk in the park, especially not since we don't run a versioned API.

Probs should start versioning though, just in general

 

[mikhi]

This one's to combat scrapers that rely on headless browser scraping. Otherwise, you're just calling an API and it isn't technically scraping. lol

Well then this change is just a bad change. The only thing it does is limit actual user. Scrapers can get past it as easy as pie. Again I don't wanna put on a thin foiled hat but sure does make you wonder.

 

[panzerkampfwagyu]

Why would scrapers even need the frontend? As I said the API has no limits. You can just carefully code your scarper to not trigger rate limits there. Maybe rotate around the IP. It really isn't hard to make a functional scraper that doesn't have to touch the Fronted.

IIRC it was mentioned here, but they are programmed to act like cunts and don't respect the sites they scrape their data from.
They ignore stuff like Robots.txt and just try to claw all the data they can without caring how it affects the site.

 

[Shrimperor]

We have temporarily reverted this since too many guests were signing up, and the auth service was beginning to degrade. We will try again later.

Really didn't think through that one huh

Well then this change is just a bad change. The only thing it does is limit actual user. Scrapers can get past it as easy as pie. Again I don't wanna put on a thin foiled hat but sure does make you wonder.

+1
This change won't do anything against bots at all

 

[panzerkampfwagyu]

On that note, there was a suggestion last year about a way to solve this particular problem, if that helps.

 

[Tony_Mushah]

On that note, there was a suggestion last year about a way to solve this particular problem, if that helps.

I wish they implemented this and replace the default girl with that sexy dex-chan from 2 years ago.

 

[panzerkampfwagyu]

I wish they implemented this and replace the default girl with that sexy dex-chan from 2 years ago.

Or... An image of Anubis together with a Security-Dex-Chan investigating whoever dares enter the holy halls of MinisterDex

 

[panzerkampfwagyu]

For real though, Anubis works by making requests a tiny bit more expensive, which doesn't change much for a normal user, but really amasses for spammers, which seems like would be the tool for the problem

 

[Tscheddov]

We have temporarily reverted this since too many guests were signing up, and the auth service was beginning to degrade. We will try again later.

i beg you. please don't try this again. it did more harm than good while not helping with the issue it was supposed to help .

doesnt the gacha game reward users with points for read chapters ?
if users just rush trough chapters for the points, doesnt this add unnecessary traffic as well ?

 

[Cruentus]

For real though, Anubis works by making requests a tiny bit more expensive, which doesn't change much for a normal user, but really amasses for spammers, which seems like would be the tool for the problem

Specifically, it works by making the first request from a given user more expensive, and the expense is client-side. The reason this stops scrapers is that they pretend to be a new user with each resource request (which they do to avoid other kinds of blocks) so they get hit with it on every load.

It's exactly the tool for IP-hopping bots.

 

[EtherealNeko]

Is this the part where I say... CALLED IT?

Well done, have a custom title to celebrate your achievement.

 

[shinya78]

Well done, have a custom title to celebrate your achievement.

Yipeee~!
Wait, there's no way for me to change it back to Dex-chan lover, dang it oh well.

For real though, the sudden influx of users signing-up flooding the servers with either malicious intent or not, anyone could've picked up on that. Scrapers gotta scrape, readers gotta read.