Add anubis proxy to stop crawlers

[shwiftyshfifty]

https://anubis.techaro.lol/

https://github.com/TecharoHQ/anubis

Still not sure if crawlers are what's causing issues with the site recently, but it tracks with other problems I've read about. Nadeko, one of the few remaining Invidious instances, recently implemented it and has become a lot more stable since then.

I don't know much about these things, but I want to let people know about it. Hopefully it helps.

Also bonus cute Anime girl-God of the Dead mascot.

mod note: merged into thread

 

[Doedoel]

First of all, what are crawlers? Are they like Trojan virus or something?

mod note: merged into thread

 

[HelperHand]

First of all, what are crawlers? Are they like Trojan virus or something?

Crawlers are like these ones, to be precise a crawler is a type of bot that search and index content from websites, one of the most common are search engine crawlers with the purpose of returning "useful" data during a search (currently too much enshitification from Google and Bing), they usually inspect and respect the sitemaps and the robots.txt and natural links from the different elements in the pages, but due to "AI", a new type of crawlers that don't respect the rules and are just trying to steal info to feed to their LLM have appeared, they like to rip everything apart and when rate limited they just change they IP and User Agent, and considering how many companies and individuals are competing with each other it has caused a lot of issues. Open Source communities have really suffer.

The first time I heard about Anubis was due to being deployed by Gnome

mod note: merged into thread

 

[Tony_Mushah]

The first time I heard about Anubis was due to being deployed by Gnome

Please don’t change the image. I want to see an anime girl when I access Gnome Gitlab.
My girlfriend also wants me to see her.

Imagine if you need to inspected by Dex-chan everytime you enter Mangadex. Can't wait for that.

mod note: merged into thread

 

[Angry_Panda]

Imagine if you need to inspected by Dex-chan everytime you enter Mangadex. Can't wait for that.

Gonna be honest - if it's OL Dex-Chan, I'm not going to object to being very thoroughly inspected.
https://forums.mangadex.org/attachments/mdex-gacha-png.19467/

More OT, I'd be interested to know if the devs have any data correlating crawlers with site slowdowns.

mod note: merged into threa.

 

[goldencircle]

https://anubis.techaro.lol/

mod note: merged into thread

 

[shwiftyshfifty]

https://anubis.techaro.lol/

mod note: merged into thread

Thanks, I didn't know this existed.

 

[77BLc9JwJGpLVARkamJx]

Probably those reader application/extension hammering image cdn/api servers or manga publishers ddos-ing every pirate site to discourage users to visit or outright take them down or just scums trying to blackmail the sites or maybe who knows those other aggregators are using mangadex backend as their image cdn network . Time for a mangadex cdn botnet lel (No idea @ home capacity right now).

Honestly putting in a seemingly cloudflare lite seems kinda eh but maybe its good since the devs know better on what fucking up the network.With the anubis thing is still new, no idea if there is a vulnerability that might be used to take down mangadex (and the people behind them) as well.No need to introduce more security headache.