Add anubis proxy to stop crawlers

[shwiftyshfifty]

https://anubis.techaro.lol/

https://github.com/TecharoHQ/anubis

Still not sure if crawlers are what's causing issues with the site recently, but it tracks with other problems I've read about. Nadeko, one of the few remaining Invidious instances, recently implemented it and has become a lot more stable since then.

I don't know much about these things, but I want to let people know about it. Hopefully it helps.

Also bonus cute Anime girl-God of the Dead mascot.

mod note: merged into thread

 

[Doedoel]

First of all, what are crawlers? Are they like Trojan virus or something?

mod note: merged into thread

 

[HelperHand]

First of all, what are crawlers? Are they like Trojan virus or something?

Crawlers are like these ones, to be precise a crawler is a type of bot that search and index content from websites, one of the most common are search engine crawlers with the purpose of returning "useful" data during a search (currently too much enshitification from Google and Bing), they usually inspect and respect the sitemaps and the robots.txt and natural links from the different elements in the pages, but due to "AI", a new type of crawlers that don't respect the rules and are just trying to steal info to feed to their LLM have appeared, they like to rip everything apart and when rate limited they just change they IP and User Agent, and considering how many companies and individuals are competing with each other it has caused a lot of issues. Open Source communities have really suffer.

The first time I heard about Anubis was due to being deployed by Gnome

mod note: merged into thread

 

[Tony_Mushah]

The first time I heard about Anubis was due to being deployed by Gnome

Please don’t change the image. I want to see an anime girl when I access Gnome Gitlab.
My girlfriend also wants me to see her.

Imagine if you need to inspected by Dex-chan everytime you enter Mangadex. Can't wait for that.

mod note: merged into thread

 

[Angry_Panda]

Imagine if you need to inspected by Dex-chan everytime you enter Mangadex. Can't wait for that.

Gonna be honest - if it's OL Dex-Chan, I'm not going to object to being very thoroughly inspected.
https://forums.mangadex.org/attachments/mdex-gacha-png.19467/

More OT, I'd be interested to know if the devs have any data correlating crawlers with site slowdowns.

mod note: merged into thread.

 

[goldencircle]

https://anubis.techaro.lol/

mod note: merged into thread

 

[shwiftyshfifty]

https://anubis.techaro.lol/

mod note: merged into thread

Thanks, I didn't know this existed.

 

[77BLc9JwJGpLVARkamJx]

Probably those reader application/extension hammering image cdn/api servers or manga publishers ddos-ing every pirate site to discourage users to visit or outright take them down or just scums trying to blackmail the sites or maybe who knows those other aggregators are using mangadex backend as their image cdn network . Time for a mangadex cdn botnet lel (No idea @ home capacity right now).

Honestly putting in a seemingly cloudflare lite seems kinda eh but maybe its good since the devs know better on what fucking up the network.With the anubis thing is still new, no idea if there is a vulnerability that might be used to take down mangadex (and the people behind them) as well.No need to introduce more security headache.

 

[Tony_Mushah]

What...??
This thread was moved from the "community project section" to the "main site suggestion suggestion".
Does that means that they will consider adding this into the main site?

 

[HDMI1]

What...??
This thread was moved from the "community project section" to the "main site suggestion suggestion".
Does that means that they will consider adding this into the main site?

Well the community really doesn't have control over the site, the devs do. So it would go here for review by the devs.

 

[panzerkampfwagyu]

Necroposting, since this here pretty much is a solution to the problem the ill-fated last attempt tried to solve.
Also, I suggest adding a detective dex-chan next to anubis-chan in the check, using a magnifying glass to investigate those entering the sacred halls

 

[JinxedMyself]

Necroposting, since this here pretty much is a solution to the problem the ill-fated last attempt tried to solve.
Also, I suggest adding a detective dex-chan next to anubis-chan in the check, using a magnifying glass to investigate those entering the sacred halls

Certainly needs some more research, but apparently you can get an unbranded version, which allows for custom CSS and images among a few things. Certainly gonna cost you (but like, custom branding is pretty cool)

 

[panzerkampfwagyu]

Certainly needs some more research, but apparently you can get an unbranded version, which allows for custom CSS and images among a few things. Certainly gonna cost you (but like, custom branding is pretty cool)

From what I understood, it only costs if you want to remove anubis-chan.
Edit: Ah, Nevermind. It does require the paid version for custom CSS

 

[AluSoN]

In my opinion, the use of Anubis is debatable. It won't be that hard or costly for those crawlers to implement an algorithm that solves the Anubis challenge on default difficulty. I'm not opposed to it, just want to point out that it's not a guaranteed solution.

What I mean is described in the "Problem" and "Numbers" sections of this article (not mine):
https://lock.cmpxchg8b.com/anubis.html

By the way, this article is 9 months old, so maybe this problem has been fixed by now.