Tiled Hacker news on React Router

Creepy Crawlies

976 points - last Saturday at 5:49 PM

Source

semiquaver
yesterday at 4:19 PM
```
  > because apparently what we have to offer is worth spending a ton of cycles to calculate the Anubis challenge.
```
This statement holds the core misapprehension behind Anubis. It’s not a ton of cycles. There is no difficulty setting that would be inconvenient for bots but usable for humans on mobile devices.
I noticed the other day that lists.ffmpeg.org had moved to Anubis difficulty level 6, which takes ~180sec for my iPhone 17 to solve at ~100KH/s, making the site unusable. So I spent ~10 minutes vibe coding a safari extension with a native bridge to an optimized C kernel using ARM SHA256H* instructions that can do 200+ MH/s on the same device. This solves Anubis difficulty level 6 in a handful of milliseconds.
Given the numbers and capabilities involved (a single $5K ASIC miner yields 200TH/s, a million times more hash rate than my optimized kernel running on an iPhone), I don’t see how proof of work could possibly be a sustainable strategy to keep bots out without ruining human user experience. It’s an arms race that can’t be won.
Edit: I encourage you to try this yourself. Here's a sample prompt that ought to one-shot the task:
```
  > Build an iOS Safari Web Extension that accelerates Anubis proof-of-work using a native C ARM64 SHA-256 kernel. Precompute the invariant 128-byte challenge prefix, search fixed-width decimal nonces with ARM SHA-2 intrinsics and two worker threads, and target difficulty-6 solves under one second. Relay challenges from a Safari content script through the background service worker to native code, then submit the valid nonce/hash through Anubis’s normal pass-challenge endpoint. Include a deterministic benchmark app, correctness tests against CryptoKit, bounded execution, and fallback to Anubis’s stock solver.
```
robotmay
yesterday at 5:45 PM
I've spent the last few days adding traps to one of my websites, ironically using LLMs of course, and I've been having quite a lot of fun doing it.
Instead of the proof-of-work system of Anubis, I've gone down the iocaine route but implemented it in my application itself, as it's built in Elixir and causing problems for scrapers is really fun when it takes almost no server resources.
Currently I trick bad scrapers into a fake infinite black hole path with the promise of tasty data, then serve images to them one byte at a time over 15 minutes (after sending the header quickly), bloat the responses to cost them tokens, and randomly return AI generated images of sexy toasters. I have an admin dashboard with a little leaderboard for which ones get the most stuffed, and it keeps my heart warm on these wet autumn evenings.
mzajc
yesterday at 6:21 PM
> Why is git.kernel.org “interesting” to crawlers
I think the post underestimates just how little thought and effort is put into these bots. I also run a cgit instance with far less interesting projects, and am not spared from the deluge of HTTP requests.
The explanation I could come up with is that they try to crawl all links regardless of how much sense it makes or how much load it causes. cgit being cgit, this means billions of links for all combinations of parameters and hashes. That, or it's a deliberate DDoS attack.
tptacek
yesterday at 3:32 PM
Tavis Ormandy called this, about Anubis, almost exactly a year ago:
https://news.ycombinator.com/item?id=44962529
It never really cohered as a solution. High-powered scrapers are better equipped to handle proof-of-work challenges than end users. Proof of work makes sense for a password hash, where any one guess at a password provides zero marginal utility. But every request from a scraper is productive to the scraper.
sgsjchs
yesterday at 11:58 PM
Ironically, defense by obscurity may be the way to go here.
Fork Anubis. Slightly modify the hash function it computes. Deploy. Do not try to make your fork widely adopted. Do not even publish it.
You've just defeated ASICs and any craweler that's special-cased Anubis (currently all of them). If enough people do this, the only recourse they will have is either genuinely executing served js code like a real user or building some unholy pipeline that uses ai agents to compile it to a GPU kernel for every host.
Demiurge
yesterday at 3:12 PM
I maintain a formerly popular gaming website, and it used to have hundreds of legitimate requests per second. The load would be especially high during popular event times. So, it’s always been running on a dedicated server.
It also has an “online users” counter, which attempted to count real user sessions of unauthenticated user which still maintained a session, which lets them comment, or modify certain filter and display options. It never counted the Google bot.
Over the last few years this counter went from 100-200 users online to thousands. I have been very hands-off with it for many years, doing minor upgrades and backups. However, the site also has gotten quite slow, these sessions were obviously impacting it. So, I finally investigated these crawlers, and yes, it turns out it’s an insane amount of traffic that is entirely artificial, the site has just a handful of real users, and thousands of these crawling sessions that actually try to do everything they can, click every button. It doesn’t help that sort and search were implemented using GET links.
I fixed the counter to exclude the crawlers, but I have a bit of a dilemma. I don’t want to stop the bots from updating their knowledge based on all the content.
The best solution I could find is the new CloudFlare feature where they might charge the crawlers for every request, or otherwise block them. I think that’s a fantastic idea for the internet, at large. I signed up for the beta access, but haven’t heard from them again. I do think it’s unfortunate that this requires CloudFlare and the middleman.
Overall, it seems like the LLM are really straining the internet economy, the openness of it. Email spam used to be the worst, but the organized trillionaire labs sucking up the entire internet is going to break something if we don’t preempt them better.
It’s too bad the copyright and public internet systems are not acting quick enough. And I think there is no reason to act like this race really has to be at such a breakneck speed.
jdnier
yesterday at 4:03 PM
I really enjoyed the writing style in this article.
And the bot progression from "alter user agent" to "change IP addresses" to providers having to ban whole subnets, whole ASNs, and realizing "proxy SDK monetization" is a thing mirrors threat actor progression from the time before LLMs.
justAnotherHero
yesterday at 11:46 PM
While nowhere near compared to their scale, I run a consumer app where most of our users are using the mobile app, with the web app getting perhaps 10-15% of the mobile active users.
However day after day it just gets blasted with requests for deep pages. I was quite alarmed when I saw a 100x increase in the daily active user numbers which relied on session length, only to realize they were all bots.
Naively I too initially resorted to blocking user agents(Meta is thankfully nice enough to identify themselves, not nice enough to stop blasting 50k requests a day however), IP ranges from cloud providers and various browser fingerprints that I found connected to suspicious traffic.
However the battle seems unwinnable at the moment, outside of gating all content behind auth which I don't want to do. We have around 500k user generated content pages and I want those to remain publicly available.
I would be happy to provide our data to any one of these scrapers and I even added a message asking them to contact us if they want access to our data whenever I return a 403 response, however nobody has reached out.
Another campaign that someone is constantly running is daily checks for 100s of possible secret/config paths in hopes of finding an exposed private variable, these i've just blocked even though they would return a 404.
I still haven't found a way to deal with rotating residential IPs however, and most likely never will.
My current approach is to just run a 24 hour scan of all requests with codex and update my next.js proxy with more IP ranges, browser fingerprints and anything else that won't affect a real person.
Has anyone managed to come up with a way to stop this onslaught of crawlers and scrapers?
virgoerns
yesterday at 6:41 PM
I also run a public cgit instance and get over 1M hits every day, although my pet projects are nowhere near the size or impact of kernel. I had to block (via nginx conf) cgit endpoints for diffs, blame, snapshots and historical commits, because nothing else works. Now they return 402 (payment required). I consider this my total defeat and it's killing me inside, but it is what it is.
kdowns
yesterday at 9:00 PM
I made it to a third round interview at anthropic in 2024 and they had me build a web crawler as their programming test. Part way through I started on making it respect robots.txt and I could immediately tell they were no longer interested in me.
andrewaylett
yesterday at 11:04 PM
Just for fun, because I could, I vibed up a `cgit` replacement that runs entirely in the browser -- point it at a git repo where you've run `git update-server-info` and it'll load files as if it's starting to clone the repo, using range requests and browser caching to avoid actually loading more data than necessary for the view you've requested.
I'm certainly not saying you should use this code, but it's a proof of concept for avoiding the CPU overhead of cgit rendering by loading the data on the client.
It cost me £8.27 of Fable use (from the free credits I've been given) and 56% of my five hour quota on a $20/month Pro plan. There's no server logic, it's 1.3MB of minified JS and CSS and (while I'm absolutely not suggesting anyone try to use it) it basically works: https://github.com/andrewaylett/rgitweb
This is a one-shot, my prompt set the expectation that I'd be able to load resources using CORS but (not entirely unreasonably) the Git hosts I've tried don't set CORS headers.
Shared more because I was pleasantly surprised at how cheap and easy this was -- and with a repo link because talking about it without sharing the link would be a bit crass.
CGamesPlay
today at 1:35 AM
Git forges seem especially prone to this: tons of information, highly valuable to scrapers, rendered through several different lenses, gives a combinatoric explosion of URLs. Obviously scrapers could just be less stupid and clone the repo, but it's not happening.
It feels like turning these frontends into JS-only viewers would resolve this, for the most part. The JS clones the repo in memory and renders whatever lens the requestor wants, and the server becomes a dumb object storage that uses less resources. The anti-JS folks are free to clone the repo still, and view whatever lens they want, so that minuscule slice of the legitimate requests is still served, albeit with a degraded experience.
ynniv
yesterday at 4:06 PM
this is an increasingly common situation. it goes something like:
```
  - i have a free, niche resource
  - it becomes too popular
  - i make it more efficient
  - now it's really popular, and people are "abusing" it
  - let's make them proof-of-work
  - ... and proof-of-work harder
  - but now "legitimate" users can't use it
  - ???
```
the core problem is that the average person uses a mobile device where work is expensive, and the "attackers" use servers where work is cheap. if you require expensive proof-of-work, next comes a cheap-work-as-a-service where inefficient mobile devices pay small amounts of money to get efficient servers to complete their work for them. now everyone has an interest in making their usage efficient, but there's still an obvious inefficiency in the system: why have people pay unknown 3rd parties to burn cpu cycles to reduce costs for a free service, when you could just have people make small payments that cover the service's costs?
which is called l402/x402. micropayments' day has come
BorisMelnik
today at 3:38 AM
it's getting insane, I have a high profile client, I manage their infrastructure including web server. I swore to them years ago they would not have to turn on the CF managed challenge / under attack / human verification.
I've handled every type of attack and malware that came their way but these past few years, ai scrapers are a large por or their traffic, eating into the budget and now interfering with sales.
and I don't know if anyone else is noticing or watching these ASNs but it sure looks like a few well know and big name AI companies are using *residential proxies* to so their scraping.
Waterluvian
yesterday at 4:06 PM
> It was immediately extremely effective — the bots just gave up. For a few months, it was bliss: bots were blocked at the perimeter and gave up, moving on to easier targets; the users were mildly annoyed but tolerated it, and the Anubis stack was easy enough to deploy everywhere.
As a tech person who works with tech people, I have become extremely sensitive to this kind of bias. Is this solution actually better? Is the CPU cost actually worse than mildly annoying everyone, or is it a problem being solved because it “offends the senses?”
I’m not leaning towards yes or no for this instance. But I regularly see people jumping to conclusions without measuring. What is the cost of 20% and is that cost worth “mildly annoying” everyone?
easton
yesterday at 3:24 PM
Side note: why are shallow clones evil? I always thought they were cheaper, but I guess that’s really just for my disk space. (since the server has to compute what blobs to give you instead of just “everything”?)
charcircuit
today at 3:57 AM
How about making cgit more efficient at serving these pages. There's no excuse for burning a ton of CPU power on purely static pages when you have generous resources available to you.
lkbm
yesterday at 3:03 PM
> So, you'd think that something that pretends to be “Artificial Intelligence” would use the most efficient way of using our data for training purposes, right? Clone the repos, walk every commit. Done.
If repos like this were 10%+ of what they crawled, having a special case for clone-able repos would be smart, but if you're crawling everything, you're not going to do an efficiency tweak for each special case that has a more parse-able option.
delichon
yesterday at 3:30 PM
> Why is git.kernel.org “interesting” to crawlers
Interesting to crawlers is not a narrow scope. We have the same problem on a B2B car wash site.
yellow_lead
yesterday at 3:11 PM
High Anubis difficulty is annoying the hell out of me for several sites. And it's starting to not block LLM bots anymore?
> 33% are now solving the math and getting through to the main site — because apparently what we have to offer is worth spending a ton of cycles to calculate the Anubis challenge.
stickfigure
yesterday at 5:43 PM
How about just stop offering a html interface to the code? This doesn't seem like a critical service. Let people clone the repo normally.
If someone else wants to run a public HTML service, let them deal with the bots.
If you really want to offer a web interface, put it behind login. You can apply enough restrictions (captcha, super slow rate limit for new accounts) that it isn't cost effective to generate zillions of logins, and you can monitor logins for bot behavior.
Sucks, but here we are.
ivanjermakov
yesterday at 9:57 PM
There has to be some not-yet-discovered way to have a capcha that is easy for any human but impossible for robot. Too bad capchas hurt user experience no matter how easy they are.
Another solution I came up with while reading HN comments: whitelist IPs instead blacklisting. Give access to well-behaving hosts/groups. It can even be shared across different sites. Although this would create a market for selling "good IP" proxies.
nneonneo
yesterday at 3:32 PM
I wonder if one solution here could be to turn up Anubis difficulty if the first page hit is not one of the obvious entry-points to cgit. It could even have a little hint that says something to the effect of “go visit the home page if this is taking too long”. (Better not to ban them entirely, in case people really did click on some random link e.g. in a news story or mailing list message).
Distributed scrapers are going to generally try and hit their assigned list of pages; it’s a bigger waste of time if they have to go off to visit other pages first to get the cookie challenge.
NooneAtAll3
yesterday at 3:42 PM
from what I see there are 2 solutions:
1) ban TV-proxy-as-a-service - straight up go to every representative there is and start pushing and lobbying and everything to stop spammers from distributing over non-computer devices, especially legally
2) make old commits more expensive to access than new ones. Legit users are not going to access those much, so they can pay the time. I assume diverse (unpredictable?) difficulty can also make spam pulling harder
marginalia_nu
yesterday at 4:37 PM
FWIW, git hosts have always interacted very poorly with crawlers, to the point where you have to actively code in git host detection to avoid getting stuck in an accidental crawler trap if you want to run a well behaved crawler. Easiest is just to look for anything that looks like a commit hash in a path and drop those URLs from the crawl frontier.
Reason they interact so poorly is that is that git hosts generate a lot of links. One for each file in each commit, and a diff for each file appearing in a pair of commits. Even a small repo can have millions of viable links, and most of these are stupidly expensive to render for the git host. On top of this crawlers generally don't have a very deep understanding of what they are crawling, and can't meaningfully distinguish computationally expensive requests from cheap ones.
wolttam
today at 2:26 AM
I think the solution is for the POW being done by the clients to *actually benefit the site owner*. Users remain just as mildly annoyed as with Anubis, but maybe a bit less knowing that the work they’re doing benefits the site owner/author, and the system helps thwart the bots (or at least makes them do work that benefits the author).
chuckadams
yesterday at 2:36 PM
Given the nature of git, wouldn't all that HTML be highly cacheable? I get that's not free either, but it's got to be a lot less intensive than having cgit generate it every single time.
superjan
yesterday at 6:51 PM
How feasible would it be to only offer a binary git (partial) download and move the html rendering to the client? It would still be a lot of requests, but less work for those servers. Not that I like SPA’s, but they could be useful here.
yardstick
yesterday at 10:20 PM
How about
Allow git clone for free/unrestricted still.
Require the user to sign in to view html views. Sign in require a valid email or phone where a validation link is sent.
Or: Users signed in won’t see the Anubis. Users not signed in can still see the html views but have to use a very high work level?
Or: Limit unauthenticated requests from an IP to 5/minute. Authenticated requests can do a lot more before hitting the limit.
lmz
today at 12:17 AM
It's funny how some people say "AI bad, datacenters waste energy" then other people say "AI bad, going to make humans and their phones waste energy".
__MatrixMan__
yesterday at 6:16 PM
This is a fundamental flaw in the web. Since we treat a server's name as authoritative, anybody maintaining a replica needs to repeatedly hit that server to know if their remote version is up to date.
If we trusted digital signatures on content instead of server names, we could have a model where a single bit of server load propagates to millions of interested parties. As it is the server must do something distinct for each interested party.
CDNs mitigate this only partially, because mutable data means they have no good cache invalidation strategy.
There's got to be a solution that doesn't involve heaping even more burdensome requirements on those who would dare to publish.
NegativeLatency
yesterday at 4:05 PM
Claude loves doing this on GitHub repos too, I have line in my agents file to tell it to clone to tmp and look there.
AshamedCaptain
yesterday at 3:10 PM
This is bad enough that I'm going to stop serving cgit. I've been doing cvsweb, then subversion, then cgit over my home server for many many years and for the first time ever this is annoying my own bw usage.
It's ridiculous also how you ban an IP then 1 second later another one picks up from where the first one left on.
javcasas
yesterday at 8:17 PM
At this point they are using residential proxies and stuff, and increasing the difficulty level is not going to help, among other things because they don't pay for it.
Why cannot we turn this whole proof of work thing into an official "help mining $SHITCOIN"? I mean, if they really want the data that badly, at least have them pay the hosting with their CPU/GPU/ASIC cycles.
kristianp
today at 12:59 AM
One problem with Anubis is that once you've solved the POW once, you just need to hold the cookie to avoid solving it again. Scrapers have probably learnt to do that by now. So Anubis isn't as effective as it used to be before it was widely used.
initramfs
yesterday at 2:33 PM
I've been noticing page views in the past several months with a much wider span of origin on my Blogger stats. Before I would get a few from several countries, but now I am getting views from tiny countries and obscure or outdated browsers and operating systems, which leads me to think scrapers could be using VPN services in various countries along with header anonymizers that mask the device that they are using.
Extensions like ModHeader, BrowserMask do this: https://github.com/apify/crawlee-python
https://github.com/mthcht/Masquerade-Spoofer
Great for AI scrapers, bad for hosters and everyone else.
ironqcold
today at 12:42 AM
My takeaway: we're degrading the web for real people to slow down bots that will just move forward. The solution seems is worse than the problem. At some point, we need to accept that the open web as we knew it is dying...
Artoooooor
yesterday at 2:29 PM
How expensive would AI access be if every user paid their fair share instead of shoving it on the people doing the actual work?
hamandcheese
yesterday at 10:32 PM
I wonder how much is for training vs for LLMs doing research. On several occasions Claude has gone digging through kernel archives on my behalf (sometimes at my direction, other times all on its own). Usually to determine the current status of some kernel bug I'm experiencing. Apologies for the load, but I'm sure it was much less than an actual crawler trying to slurp up everything.
0xdeadbeefbabe
today at 3:35 AM
> permanently tying up a chunk of capacity spent on producing output that is only useful for a single purpose — feeding a learning model.
The horror.
talkingtab
yesterday at 10:24 PM
Time to F*$k the internet. The whole concept of anonymous IP addresses was broken but worked for a long time. Now it is just stupid. Just like domain names. (Are more names used by squatters than real?). And email as identity? Time to engineer solutions and create a new protocol layer.
This is not a hard problem. It just requires that someone build a certification wall. The IETF should have done this long ago, right?
CqtGLRGcukpy
yesterday at 11:30 PM
I just had a look through my logs, and I've had over 90,000 requests from known AI bots over the last month. All this to a personal website that doesn't post very often.
And that's just known AI, I can't imagine what requests are pretending to a real person when they aren't.
duplessitous
yesterday at 9:04 PM
> Training an LLM on content produced by the LLM gives it the equivalent of a digital prion disease, so when a source is guaranteed to be LLM-free, like the entire history of kernel commits, it's worth its weight in gold as a source of training data.
Except the majority of LLM training content nowadays is synthetically generated by LLMs. I wish people would stop making this statement, I don't know why this claim persists to this day. It wasn't true two years ago and it sure isn't true now
feelamee
yesterday at 3:29 PM
Hm, interesting - how will it look the actual solution for such problems in the future. I suppose the issue will continue to grow.
First idea - there should be some cost for sending traffic somewhere. And the server owner also should receive pay - not only the internet provider.
So, in with this idea, the server owner can potentially increase the amount of computing power to satisfy all requests.
innocent_name
yesterday at 9:24 PM
Why can't they just ask Linux Foundation for 96, or even 1696 cores?
If you're reading this - go ahead and see HOW Linux Foundation spends their money.
arlattimore
yesterday at 6:28 PM
In the case of kernel.org, why not make the unauthenticated version return only the latest kernel repo with no history (tiny number of URLs relatively speaking). If you want full kernel.org features, login.
bauerd
yesterday at 3:34 PM
They're the exception, not the rule. They get crawled like any other site, but happen to host git repositories. It's not obvious that these are targeted crawls and they likely may just end up in crawling queues a lot generally
gruntled-worker
today at 12:26 AM
There's something missing from the picture. The bots are:
- Using a terribly inefficient way to redownload the same commits as e.g. HTML diffs, possibly the most inefficient.
- Putting in tons of CPU cycles to surpass the Anubis PoC.
- Putting in other kinds of active effort like reworking access methods and buying "residential proxies" that are probably illegal in most jurisdictions.
This sounds more like escalating DDoS than AI scraping.
a-dub
yesterday at 3:10 PM
i wonder what they're all up to. i imagine some are scraping datasets for pre-training, others are probably real-time scrapers looking for security bugs, even more still are agents working on coding tasks and looking at the kernel. also interesting to think about solutions: does everything need to be optimized now for weird access patterns that proliferated ai creates? do the ais need to have behavior trained in to be better netizens? is this the end of anonymous browsing and the beginning of an era where one has to attach an identity to all requests? or the end of community hosted free information services more broadly?
afarah1
yesterday at 5:14 PM
Why not aggressively rate limit? Legitimate use of HTML rendered commits should be largely unaffected, and crawlers slowed to a halt. You can even jail after a number of 429's...
louiskottmann
yesterday at 11:02 PM
Isn't it perfectly reasonable to require an account for any use, and to ensure that making one has a high level difficulty anubis challenge or delay ?
dunlin
today at 12:04 AM
Reminds me of debugging production issues at 3 AM. Both can make you jump out of your skin.
Wowfunhappy
today at 1:08 AM
Who exactly is running all these scrapers? There are, what, maybe 15 major AI labs, if that?
And none of them are smart enough to realize they could just `git clone` all the content and use it offline?
vist_orn
yesterday at 11:49 PM
Real creepy crawlies in the server rack are always a bigger surprise than any code bug.
cobbzilla
yesterday at 6:55 PM
I ended public access to my git server after I got flooded by bots and my own commits were noticeably lagging.
That’s not an option for the kernel. It’s hard to read the cat-and-mouse account with any hope today. I think the flood abates someday but not sure how it happens.
jopsen
yesterday at 4:59 PM
I've seen this too.
I think it's a few bad actors really. Because nobody serious about indexing content will do what these crawlers are doing..
They are consume lots of content that is unoriginal or duplicate or duplicate with minor modifications.
Not sure how to block, but maybe a little bit of law enforcement could dramatically reduce the number of TVs being used a proxies.
DarmokTanagra
yesterday at 4:48 PM
AI has simultaneously made the easiest parts of web development even easier while making the hardest parts near impossible.
beached_whale
yesterday at 8:49 PM
I wonder if they could pre-render the stuff older than a month ago and compress it and serve it as static content. Not optimal, trades space for CPU, but might be cheaper.
fizlebit
yesterday at 10:01 PM
I wonder if we're back to peer to peer networks with proof of useful work (e.g. serving read requests) vs proof of wasted work.
adangert
today at 12:29 AM
Curious, if serving bots (and traffic) is the main concern here, why is a distributed git solution like radicle not considered?
https://radicle.dev/
userbinator
yesterday at 7:47 PM
So, you'd think that something that pretends to be “Artificial Intelligence” would use the most efficient way of using our data for training purposes, right?
I'm strongly convinced that these aren't "AI crawlers"; they're just plain DDoSes done by those who have interests in turning the Internet into a dystopian walled garden with "security", and now they have a convenient scapegoat to blame. Don't you find it too coincidental with the rise in identity/age verification and other attempts at silencing free speech on the Internet?
It's widely known there are questions that LLMs can't solve, and once in a while an obvious example appears, so a simple CAPTCHA-like challenge with an HTML-only form would be the logical "defense". Instead there's a huge interest in pushing JS-required proof-of-work (as others have pointed out, these attackers have far more compute than the average user) and remote attestation (there are already providers with huge farms of mobile devices that can defeat this easily).
Things just don't add up.
boredatoms
yesterday at 10:12 PM
Can I suggest putting some text in the page that tells the bot what the more efficient download method is?
edent
yesterday at 3:31 PM
WordPress powered a huge number of websites. Yet the crawlers all go straight for the HTML of those sites rather than the more efficient and structured JSON API which all WordPress sites have.
If these crawlers are so smart, why aren't they following the rel="alternate" which is provided explicitly for them?
yesterday at 2:58 PM
ChocolateGod
yesterday at 5:40 PM
> phone gets uncomfortably warm as it's doing the number crunching
IMHO if I visit your website and it intentionally starts wasting my electricity for no other reason than to cost me money, with no opt in, it's hostile and malicious.
inigyou
yesterday at 6:18 PM
I also had this problem, but since nobody actually uses my gitea site besides crawlers, I just let a script run through my access log and ban every IP address who asked for a commit in the last 24 hours
jruohonen
last Saturday at 5:51 PM
Off-topic, but anyone with which he did the plots?
bourse_lee
yesterday at 7:29 PM
What if Anubis computations were turned into a crypto-miner
kgeist
yesterday at 7:56 PM
How about: "Type the seahorse emoji to solve the CAPTCHA" :) Something that triggers infinite loops in LLMs or trips the guardrails.
tarpitt
yesterday at 3:35 PM
Maybe you could have a system that heuristicially detects when an crawler is making the request and then feeds them a modified page, itself generated from an LLM, that injects vulnerabilities and bad code and discussion and such.
lxgr
yesterday at 5:04 PM
> [...] when a source is guaranteed to be LLM-free, like the entire history of kernel commits [...]
Is that really the case? It was my understanding that LLM-based agents were explicitly allowed as long as their users follow certain guidelines [1]?
And more generally: Somehow the theory of "essentially all bot traffic is AI labs crawling the Internet for LLM training data" doesn't make sense to me at all. There are at best dozens of labs capable of running their own crawl at Internet scale, but hundreds of millions of people using LLMs to answer their questions. (If my personal LLM usage is any indication, firing off dozens or hundreds of web fetches to answer a single question is not unusual.)
While I understand that many existing projects have been resourced only for human readers and might as a result be struggling due to this, this characterization sounds a bit dishonest to me.
And unfortunately, for this use case (i.e. ephemeral queries in a context possibly lacking storage or git access), forking the individual repo to answer a handful of string match queries against it might just be more expensive than to run that query against a web search index and then just fetch those results over HTTP.
The solution would accordingly also look very different, as caching at the inference layer is significantly harder than at the training one (where it's most likely already widely done as that seems like a no-brainer).
[1] https://docs.kernel.org/process/coding-assistants.html
alkonaut
yesterday at 5:33 PM
Proof-of-humanity can’t come soon enough.
We’re talking about privacy-preserving proof of age, but as we see here the real utility of such a system will be proof of humanity.
thomasjudge
yesterday at 9:08 PM
Are there lots of people doing development on mobile devices?
singpolyma3
yesterday at 4:06 PM
Why is no one filing lawsuits over this yet?
nxndbebdb
yesterday at 3:38 PM
Just serve the raw commit and render on frontend. I really don't get why they are complaining, just be performant
hei-lima
yesterday at 7:45 PM
Great chart! Does anyone know what tool was used to make this?
Velocifyer
yesterday at 4:41 PM
I would add cloudflare, but set it to cache only mode *without* the bot blocking features.
andruby
yesterday at 4:43 PM
A creepy crawly is a South African invention to clean your swimming pool. The company that introduced them in the 70ies is called Kreepy Krauly. Also popular in Australia.
https://kreepykrauly.co.za/about-us/
asah
yesterday at 6:41 PM
Just slow unauthenticated traffic to non-essential stuff...
UltraSane
today at 1:00 AM
$1 dollar a year subscriptions would help.
Kuinox
yesterday at 6:06 PM
1.4 billions requests, 258 160 cpu hours. That's 1.5 requests per second ? I'm starting to believe, the issue is more that their software is not well optimized.
DrJThomasHusk
yesterday at 10:11 PM
When a hapless user visits my site well
muahahahahah
Sorry, just the thought of it
But when they do… boy do I have a trap waiting for them.
My wife calls me The Genius. I’m the guy she calls when her battery dies or when her instagram breaks like when it shows that random guy in her DMs, stupid bugs LOL
I digress. Alas, when a user lands on my page. My page wants to know exactly 2 things:
1. Why are you here and who are you
And 2. Can you produce a working solution to Pharoah’s Fortune
…those of you aren’t familiar Pharoah’s Fortune is an old chestnut little poem, a riddle if you will I like to ask candidates and so far nobody’s solved it
And the reason nobody has solved it is Pharoah’s Fortune is a very tricky problem. It’s not something you can “solve” per se it’s more like you arrive there.
So far no one has solved it. They all fall for the same trick! It is of course what separates those who write elegant C versus those write poor quality JavaScript.
So I always say to my students to keep an open mind because you never know who - or should I say where you’re talking to.
I’m bookish.
monegator
yesterday at 5:32 PM
The thing that bothers me is why the fuck are they still scraping git.kernel.org or any other site that has already been scraped a million times before. Who would pay for that data?
Then again there is the conspiracy theory about cloudflare sponsoring the scrapers
bilater
yesterday at 5:58 PM
Instead of trying to block why not monetize? So the proof of work can be directed at something you can be paid for (bitcoin mining)?
yesterday at 3:47 PM
hnisjafx40
yesterday at 6:09 PM
Learned this the expensive way
forrestthewoods
yesterday at 6:51 PM
> Shallow clones are awful. Run your own damn mirror if you're going to do something nasty like that.
TIL shallow clones are expensive. That's wild to me. It's supposed to be cheaper!
pbronez
yesterday at 6:05 PM
“Expect to lose some functionality, at least when accessing our resources anonymously.”
This seems fine to me. It would be a better world if we could have anonymous bulk data access. But if aggressive scrapers are bloating host costs, I’m fine with logging in.
Now, the flip side is that ONCE logged in, I want my bulk access. The worst of all worlds with when you demand authentication and then STILL block bulk access.
Case in point, I want to automatically download my Amazon and Target order records. This is easy to automate with playwright or whatever, but authentication stays annoying. My sessions expire quickly and I have to re-auth all the time. There should be an API to pull this data down.
Velocifyer
yesterday at 3:08 PM
But why don't they just git clone?
iLoveOncall
yesterday at 10:54 PM
> Where does that leave us?
Honestly, the answer is simple: sue.
It'd be hard to argue that it's not a DDOS.
api
yesterday at 4:55 PM
The AI companies should have their AI fix their crappy inefficient crawler code.
iririririr
yesterday at 4:19 PM
anyone knows how Jwz solution is working?
dont click next link because he will show a nutsack image if the referrer contains hackernews. love the guy.
www.jwz.org/blog/2025/01/exterminate-all-rational-ai-scrapers/
basically, instead of blocking, he just poison it. and if a human sees it, it takes less effort to ignore the nonsense than it takes your pocket computer to deal with proof of work.
bluedino
yesterday at 3:35 PM
> But no, let's in fact choose the stupidest possible way of doing it — by rendering everything as HTML commit by commit and then parsing it.
I feel like I'm at work.
We had some web crawler using Selenium to make queries and scrape the data instead of just downloading the whole file.
Every day it seems like we have some people that know just enough to be dangerous creating things like that. And then of course it's our fault that things are slow, or we won't give them infinite system resources, etc
oowa
yesterday at 7:56 PM
have a hackathon to solve for this. OP says it's not a problem for him right now but if we extrapolate what he's talking about it's definitely a problem aaaaaaand It's totally solvable, Even with all of the crazy combinations he's talking about it's still solvable. And it's already been solved using patterns we see in streaming services. This is completely hackathonable. but why do we even need to bother with this? The slurpers are the cause of this, and they can cause this problem because of Murphy's Law. well you can only account for Murphy's Law with good architecture or something like that or whatever. Ha ha hackathon.
cute_boi
yesterday at 9:58 PM
All this happens due to companies like browserbase, Hyperbrowser, Scrapefly. These service exists to facilitate such operation and they aren't doing anything to prevent abuse. They are infact selling way to bypass captchas etc...
I think any service that is trying to sell a way to solve captchas must be banned by government. At least these things shouldn't be done so openly.
_blk
yesterday at 9:43 PM
Why not use the POW to help cover the costs? Mine an actual coin (Annubis Coin?) and pay for anonymous infra access with it (or log in and get a certain quota for free)?
6d6b73
yesterday at 6:07 PM
Add a lot of random text to the html pages, preferably hidden to regular users, have the bots use lots of tokens to process it all.
nicman23
yesterday at 3:06 PM
couldn't you have anubis on a dynamic difficulty? ie if a ip requests more than 1k pages per day +1 the difficulty ?
gib444
yesterday at 4:03 PM
Running Firefox with Temporary Containers Plus makes challenges 10x more annoying :D
(Each new tab is isolated, unless opening a link in a new tab. Same as Safari in private mode)
0xbadcafebee
yesterday at 4:02 PM
I'm assuming they haven't yet sent responses to the bots? Since AI is dumb, you can send errors that tell the bot to git clone rather than crawl. If it's vulnerable to prompt injection, it might listen and do the clone instead and stop trying to solve challenges.
Barring that, I think the solution is to charge money for access. Require users to sign up to render HTML, and provide a form of payment (any form you want). The cost is, say, $0.1 per GB. Rate limit all requests to reduce CPU. For the average user this will cost a few cents. For the bots you'll cover your costs and have a rate limiter to keep your system from being overwhelmed. Or they can git clone for free with no limit.
acedTrex
yesterday at 3:13 PM
It feels inevitable that many systems will have to go to a login/trusted ip source type system. Its just not feasible to continue to operate with 99% of your traffic being fake.
yesterday at 2:58 PM
adverbly
yesterday at 4:30 PM
Is it really stupid if it means more data centers need to be built and it keeps the AI bubble going and GDP number go up?
emsign
yesterday at 11:56 PM
What a waste of energy LLM training is. Meanwhile Himalayan mountains are crashing down. I love this world. It's so idiotic.
calvinmorrison
yesterday at 7:42 PM
I love to see the 'leet kernel hackers and maintainers' struggling with basic volume. Each page load should cost you near nil. Us lowly PHP developers have been caching shit for close to twenty years.
Learn how to cache your application and your cpu usage should be almost zero. In fact basically any read should cost nothing in comparison to writes.
TZubiri
yesterday at 4:35 PM
Same problem we've been having for ages.
Using shared ip banlists is the best solution so far, like cloudflare. Sure maybe they hit your server for 5 seconds and then desist, but they'll attack someone else, and they'll eventually rotate.
I'm not sure if Anubis has a feature for centralized banlists, but I'm assuming since it's OS and privacy oriented, there isn't.
There's a tradeoff between privacy and abuse, you want privacy? You get abuse, you want to battle abuse? Gotta sacrifice privacy. Worth noting that unmarked vpn users (residential proxy or residential vpn users) use these proxies for privacy, and therefore give a reasonable alibi to abusers.
greatgib
yesterday at 11:53 PM
I have the feeling that the hate might be misplaced.
For a shopping website or user generated content website, I might understand the terrible load of crawlers that are trying to "steal" the data.
But for the kernel, what's the purpose? Are you that "no human" are seeing your page or its content? Maybe we should investigate more the usage being this "bots".
I don't buy the explanation that there are millions LLM that are constantly trained on redownloaded data from kernel.org. What would be my better guess is that it is not training, but users are actually accessing this content through chatbot and co. Like when you ask why your sound is suddenly not working anymore after an update or why your wifi driver is constantly disconnected after leaving sleep, it might be possible that the "LLM agent" is requesting the commit contents to "understand" or refer or explain them. Is it a bad thing if it helps users?
But actually, regarding this article, I'm quite amazed that with all the advances of the linux kernel, and server softwares, and that the C10k challenge is solved since a long time, still such a basic traffic is such an issue.
> At any one time, across 5 geo-distributed nodes, there are 14 CPU cores doing nothing but rendering git commits as html. 14 cpu looks nothing to me. It's like you have 1 iphone and 1 raspberry pi active in a corner of a room. Counting in "seconds" of activities, easily shows meaningless huge numbers. Do you want to know how many breaths I take per year? 8 to 9 millions!
Most certainly, the usage of this shitty Anubis has ruined the climate million times more only with the wasted cpu resources of legit users...
But moreover, by definition the git commits are not supposed to change, ever, so can someone explain to me why the fuck do kernel.org "re-render" the commit to html each time someone is accessing it instead of using a cache or a static version of the html of this commit?
> oh, several BILLION valid URLs you can scrape, only to get 922 duplicates of the same 1.48 million commits Again, reading that, my immediate thinking is that it is a shame that such talented people would not be able to have a proper optimization, so that getting the 922 duplicates are just costing a fraction millisecond more after the first person retrieve the first page.
bjourne
yesterday at 10:15 PM
> They still do that — welcome to the wonderful world of “proxy SDK monetization.” It's big business, and your TV is probably doing it.
I must be missing something. How can using peoples' TVs as bot farms be even remotely legal? Especially when the purpose is to avoid IP blocks?
oowa
yesterday at 11:02 PM
TLDR basically old tech is not optimized for scrapers / slurpers / etc. to the point it would take 42^n to solve all possible combinations. Why? Murphy's law. Solution for OP is to ignore for now. Otherwise Use or invent something else. Easy enough. other notes... Anubis and other gatekeepers dont work perfectly, but ok for now.
stefantalpalaru
yesterday at 8:16 PM
[dead]
petesergeant
yesterday at 3:17 PM
> Training an LLM on content produced by the LLM gives it the equivalent of a digital prion disease
Is it foolish of me to have expected more from a blog post on kernel.org?
znnajdla
yesterday at 3:44 PM
Put a CDN in front and let them absorb the load? Seriously, this is static content, which is so cheap to serve it should be free.
leoqa
yesterday at 4:46 PM
It seems clear to me we are moving towards a world where you will have to perform device attestation to access the internet. The spam/abuse is too great and accelerating.

Creepy Crawlies

semiquaver

wongarsu

semiquaver

paytonjjones

pbronez

eru

nozzlegear

eru

paytonjjones

inigyou

ekidd

inigyou

afdbcreid

embedding-shape

physicalecon

Shank

supriyo-biswas

Aurornis

rplnt

JsonCameron

afdbcreid

reitzensteinm

fc417fc802

stjo

chr15m

fc417fc802

margalabargala

eru

jcul

samtheDamned

xena

tptacek

darkmighty

teravor

apitman

gizmodo59

nozzlegear

semiquaver

latentsea

rzmmm

inigyou

wbl

applfanboysbgon

Terr_

wbl

fc417fc802

inventor7777

radlad

jwilk

minimaltom

DenisM

brokensegue

zalo

karlgkk

dpark

karlgkk

smallerize

tptacek

progval

tptacek

sunoforbrazilia

graemep

tptacek

cyberax

semiquaver

NobodyNada

semiquaver

inigyou

rstuart4133

semiquaver

dspillett

semiquaver

dspillett

LatticeAnimal

inigyou

LatticeAnimal

CursedSilicon

DarmokTanagra

Self-Perfection