Long term strategies to address bandwidth issues and running costs: MangaDex@Home + Affiliates + BTC
- Thread starter ixlone
- Start date
- Status
- Joined
- Mar 25, 2020
- Messages
- 1
The problems at SEA region is computer hardware is too expensive, Internet service provider is not reliable and not many people can set up complicated server at home... From SEA
If it's any consolation, the hardware requirements is not very high for a server of this type. I am able to run sadpanda @ home using a sub $40 AUD Raspberry Pi3 Model B and it is using SD card and a 64GB USB Drive to store the cache files. I set the max upload speed to 1mbps and it has been running fine for the last 3 years.
My entire setup costs less than $85.
Bandwidth wise it's a different story, but with enough peers on the network and tweaking how each peer contributes to the manga galleries it can alleviate the strain for each particular peer (meaning you won't need a very high upload speed to participate).
My entire setup costs less than $85.
Bandwidth wise it's a different story, but with enough peers on the network and tweaking how each peer contributes to the manga galleries it can alleviate the strain for each particular peer (meaning you won't need a very high upload speed to participate).
Fed-Kun's army
- Joined
- Mar 9, 2019
- Messages
- 757
@ixlone
Okay, important question.
Encryption of P2P traffic: In traditional P2P applications such as Deluge (created by students & faculty at Oregon State University's Department of Computer Science & Engineering) offer the ability to encrypt the datastream with a form of end-to-end encryption, such that it makes deep packet analysis by ISPs and others extremely difficult.
Since I presume some form of hash (unique identifier) will be associated with each image file—how is data encryption for the traffic set to work, given US ISPs like Comcast are notorious for throttling content?
In the US. with the destruction of Net Neutrality, all US traffic can be throttled—some ISPs are notorious as we generally don't have municipal internet service providers, but rather private corporations [insert rant]. Those rare exceptions are glorious!
I have Comcast Business at my house, which gives me priority speeds with ~15–25 Mbps up and ~75–100 Mbps down. I wager you lot want fiber connections with equal speeds both ways.
Since localized traffic in the US would help reduce latency and improve service, what encryption methodologies have you implemented with the P2P fetching service to help circumvent deep packet analysis (such as end-to-end encryption)? If possible, while by no means perfect, any means to encrypt the streak would certainly help make things harder for those who would wish to harm hosters as well as those ISPs who go throttle crazy.
Anyhow. Thanks for your hard work. And thank you to everyone who participated in this epic level of design work! You are ALL fracking awesome! [Hands out boxes of EPIC Cookies.]
Estimation of Image Cache Size: Do we have a running estimate of the total size of the image cache? How many TBs of image data do we have currently?
Many thanks for answering my questions! :3
Okay, important question.
Encryption of P2P traffic: In traditional P2P applications such as Deluge (created by students & faculty at Oregon State University's Department of Computer Science & Engineering) offer the ability to encrypt the datastream with a form of end-to-end encryption, such that it makes deep packet analysis by ISPs and others extremely difficult.
Since I presume some form of hash (unique identifier) will be associated with each image file—how is data encryption for the traffic set to work, given US ISPs like Comcast are notorious for throttling content?
In the US. with the destruction of Net Neutrality, all US traffic can be throttled—some ISPs are notorious as we generally don't have municipal internet service providers, but rather private corporations [insert rant]. Those rare exceptions are glorious!
I have Comcast Business at my house, which gives me priority speeds with ~15–25 Mbps up and ~75–100 Mbps down. I wager you lot want fiber connections with equal speeds both ways.
Since localized traffic in the US would help reduce latency and improve service, what encryption methodologies have you implemented with the P2P fetching service to help circumvent deep packet analysis (such as end-to-end encryption)? If possible, while by no means perfect, any means to encrypt the streak would certainly help make things harder for those who would wish to harm hosters as well as those ISPs who go throttle crazy.
Anyhow. Thanks for your hard work. And thank you to everyone who participated in this epic level of design work! You are ALL fracking awesome! [Hands out boxes of EPIC Cookies.]
Estimation of Image Cache Size: Do we have a running estimate of the total size of the image cache? How many TBs of image data do we have currently?
Many thanks for answering my questions! :3
- Joined
- Nov 7, 2018
- Messages
- 21
@zhiayang
if this will work like h@\h the IP will be assigned as soon as you connect (and then you will get a domain asigned)
so this will even work on dynamic IPs
I assume m@\h will use the same mechanism.
@TheDragonLord
it's not a real p2p-system like torrent. It's like torrent if you see part of chapters as part of the mangadex system. But it's more or less a master-slave-client setup and not a real p2p.
It's more of a star-system; Clients will ask your server. If you don't have the image, the m@ h client will request it from the mangadex-servers.
edit
added a server in SEA
if this will work like h@\h the IP will be assigned as soon as you connect (and then you will get a domain asigned)
so this will even work on dynamic IPs
I assume m@\h will use the same mechanism.
@TheDragonLord
it's not a real p2p-system like torrent. It's like torrent if you see part of chapters as part of the mangadex system. But it's more or less a master-slave-client setup and not a real p2p.
It's more of a star-system; Clients will ask your server. If you don't have the image, the m@ h client will request it from the mangadex-servers.
edit
added a server in SEA
Good job team, this is sorely needed. Sadly first implementation isn't fantastic (as expected of a v1 
). Main thing desperately needed is a image loading speed ranking index, like e-h has. I've experienced a number of very slow page loads since this has gone live, and most have come from mangadex.network URLs. I note that the client already reports back on success or failure, so perhaps this should be expanded to also cover page load speed?
I guess to avoid bad data from very slow clients, this should be indexed against the clients load speed from the main image servers... but I don't know how you'd filter out influence of overloaded image servers from that. Still, an imperfect measure would be better than none.
Kinda glad it's not my job lol.
EDIT: Oh and data-saver being on lead to images 404ing until it was turned off.
). Main thing desperately needed is a image loading speed ranking index, like e-h has. I've experienced a number of very slow page loads since this has gone live, and most have come from mangadex.network URLs. I note that the client already reports back on success or failure, so perhaps this should be expanded to also cover page load speed?I guess to avoid bad data from very slow clients, this should be indexed against the clients load speed from the main image servers... but I don't know how you'd filter out influence of overloaded image servers from that. Still, an imperfect measure would be better than none.
Kinda glad it's not my job lol.
EDIT: Oh and data-saver being on lead to images 404ing until it was turned off.
- Joined
- Jan 22, 2018
- Messages
- 10
NIce to see this idea made it over here. I used to run H@H for years but never reinstalled it when the server it was on died. With my 50/50 connection, H@H never cared and worked fine however the requirements here mean I won't be setting that up. Probably for the best anyway since my UTM/network setup changed quite a bit since those days.
Either way, from someone with a massive backlog, thanks for helping to speed up the old chapter load times.
Either way, from someone with a massive backlog, thanks for helping to speed up the old chapter load times.
Fed-Kun's army
- Joined
- Mar 9, 2019
- Messages
- 757
Da!m! @izuzu I that fiber channel you got there? 😅😂🤣 Yet again, the US general public is screwed over thanks to a failure to implement municipal internet (we also have the whole massive distances to cover, but even then, the numbers don't add up. =P)
What do you have to throw down monthly for that fat pipe? 😅😂🤣 Here, Comcast is charging me nearly $200/ month for 75/25 Mbps with priority service & no data cap. That looks like what some of those small towns with municipal internet get out in the boonies! 😅🤣🤣 I saw Bellevue, Iowa advertising for like $125/month, 1Gbps/1Gbps fiber municipal internet with no data cap. 😅😂🤣
What do you have to throw down monthly for that fat pipe? 😅😂🤣 Here, Comcast is charging me nearly $200/ month for 75/25 Mbps with priority service & no data cap. That looks like what some of those small towns with municipal internet get out in the boonies! 😅🤣🤣 I saw Bellevue, Iowa advertising for like $125/month, 1Gbps/1Gbps fiber municipal internet with no data cap. 😅😂🤣
- Joined
- Nov 7, 2018
- Messages
- 21
hey @Holo,
will the statistics be per region?
E.g.
#servers
#Cache-Hits
#Cache-Misses
available bandwidth
used bandwidth
I can probably provide some more slices but i would try to setup them in places where they are needed the most.
(though SEA is quite a problem; EA (like JP or HK) is more or less doable)
Up till now there seems to be some rather slow image-servers here (if they are availbale) :/
e.g. the one from island is quite slow for me, but somehow tollerable, I would say
and some ovh-server just doesn't answer at all.
@frogcrush
This can actually happen - but normally publisher will contact the page itself - not the individual host.
But there still is the possibility for that, yes.
will the statistics be per region?
E.g.
#servers
#Cache-Hits
#Cache-Misses
available bandwidth
used bandwidth
I can probably provide some more slices but i would try to setup them in places where they are needed the most.
(though SEA is quite a problem; EA (like JP or HK) is more or less doable)
Up till now there seems to be some rather slow image-servers here (if they are availbale) :/
e.g. the one from island is quite slow for me, but somehow tollerable, I would say
and some ovh-server just doesn't answer at all.
@frogcrush
This can actually happen - but normally publisher will contact the page itself - not the individual host.
But there still is the possibility for that, yes.
Group Leader
- Joined
- Apr 26, 2018
- Messages
- 392
Apologies if this has been answered already, but how do we sign up with a dynamic ip? The sign up page request a IP address and doesn't allow a FQDN for that field. Do we just enter our current IP address for the request and then in the actual server(?) enter the FQDN?
- Joined
- May 29, 2019
- Messages
- 1
Is there a reason why a homebrew MangaDex mirroring is better than an IPFS version?
(see "InterPlanetary File System" for official site and wikipedia)
Sure, it'd require an IPFS client for the ipfs:// URL support in browser, but the technology is well-tested and you get a fully distributed P2P website with a single source of truth, where MangaDex admins serve mostly just the hashes of content instead of many copies of content.
(see "InterPlanetary File System" for official site and wikipedia)
Sure, it'd require an IPFS client for the ipfs:// URL support in browser, but the technology is well-tested and you get a fully distributed P2P website with a single source of truth, where MangaDex admins serve mostly just the hashes of content instead of many copies of content.
- Joined
- Nov 7, 2018
- Messages
- 21
@KazuyaKan
not sure why they need this - probably to locate your region or because they have to update the DNS-entries manually.
this should also work with a dynamic IP - but port-forwarding is required, since it will only work with ipv4
This is also a problem with the h@/h - no valid ipv6 Support.
the sources of the client are actually quite okay - so changing to v6 shouldn't be that much of a technical problem but a location-based problem.
@Holo
do you use the same funkctionality as h@/h ? so using wildcard and requesting the cert & key from some servers?
This should actually get rid of the problem with the certificate-generation since you can now add multiple servers in one subdomain - e.g. *.<some_id>.mangadex.network
though this will require a bit of downloading and checking certs from some system etc. as well as dynamically add/remove DNS-entries.
but this doesn't look like a big problem here. You can probably do a complete automatisation - even with the api of cloudflare
not sure why they need this - probably to locate your region or because they have to update the DNS-entries manually.
this should also work with a dynamic IP - but port-forwarding is required, since it will only work with ipv4
This is also a problem with the h@/h - no valid ipv6 Support.
the sources of the client are actually quite okay - so changing to v6 shouldn't be that much of a technical problem but a location-based problem.
@Holo
do you use the same funkctionality as h@/h ? so using wildcard and requesting the cert & key from some servers?
This should actually get rid of the problem with the certificate-generation since you can now add multiple servers in one subdomain - e.g. *.<some_id>.mangadex.network
though this will require a bit of downloading and checking certs from some system etc. as well as dynamically add/remove DNS-entries.
but this doesn't look like a big problem here. You can probably do a complete automatisation - even with the api of cloudflare
- Status
Similar threads
- Suggestion
