How are you faring against the apparent horde of AI crawlers?

[shwiftyshfifty]

I recently read this blog post:

https://drewdevault.com/2025/03/17/2025-03-17-Stop-externalizing-your-costs-on-me.html

and it made me wonder how my favourite site was faring against the vermin tide.

There was another blog post about this one that also highlighted issues many, many other sysadimns are facing; particularly open source/non-profit organizations, but I'm too eepy to find it rn.

 

[Tony_Mushah]

I recently read this blog post:

https://drewdevault.com/2025/03/17/2025-03-17-Stop-externalizing-your-costs-on-me.html

and it made me wonder how my favourite site was faring against the vermin tide.

There was another blog post about this one that also highlighted issues many, many other sysadimns are facing; particularly open source/non-profit organizations, but I'm too eepy to find it rn.

That's probably can explain why we got this issue going on lately.
Not gonna lie, MangaDex is working fine for me here in Madagascar (site and images included).
Maybe it is because AI Crawlers are overloading shards (or nodes) in certain regions for training data.
And with democratization of The Model Context Protocol, it will be more worse in the coming months.

 

[Spaghetti7173]

That's probably can explain why we got this issue going on lately.
Not gonna lie, MangaDex is working fine for me here in Madagascar (site and images included).
Maybe it is because AI Crawlers are overloading shards (or nodes) in certain regions from training data.
And with democratization of The Model Context Protocol, it will be more worse in the coming months.

Wait, you're actually in Madagascar?

 

[robertelch]

Mangadex issues temp bans when too many requests get send from a network in a short time period. The temp bans can get extended when even more requests are made while it's still active. I'd wager that helps at least a bit

 

[Tony_Mushah]

Mangadex issues temp bans when too many requests get send from a network in a short time period. The temp bans can get extended when even more requests are made while it's still active. I'd wager that helps at least a bit

It "may" help but that sure doesn't stop those AI crawlers to use perfect IP and user agent rotation to avoid the temp bans (and overloading the servers along the way).
And since those crawlers are backed by big tech companies...