CAPTCHAs can still detect AI agents
54 points - today at 3:57 PM
SourceJust ask, "I need to wash my car. If a carwash is 50 ft away should I walk or drive?"
Cider9986
today at 5:46 PM
CAPTCHAs are great. Exploiters get around them with proprietary anti-detect browsers and unethical residential proxies, while privacy browsers and affordable privacy VPNs get blocked and shadowbanned to death.
Fingerprint.com, while not a CAPTCHA, gives you +3 suspicious score just for using privacy settings like adblock on your browser. This makes it harder to sign up for any sites that use fingerprint.com.
https://github.com/CloakHQ/CloakBrowser is a good anti-detect browser as well as CAPTCHA bypass which is honestly fun to use coming from privacy browsers because every site just works and captchas get solved.
Exploiters might get around them in isolation but they are easily caught at scale due to the opportunity cost being less than the cost of creating unique behaviour over many containers.
Cider9986
today at 9:04 PM
That's cool your solution is privacy focused.
Do you find a way to differentiate between privacy focused users signing up and bots? Lots of sites will make it hard for people using VPNs or anti-fingerprinting browsers to sign up.
wonkyfruit
today at 6:22 PM
I had to do a Captcha the other day, and the letters looked awful, so I clicked the speaker for an audible Captcha instead. I was even more horrified. The sound was almost painful. Sharp noise blasting as a high pitched tinny voice bellowed numbers at me. I honestly don't know how blind people use the internet these days with such blockers in place, and that's kind of sad. The cookie banners, the captchas and the bots and laws that made both appear have kinda en$hittified humanity's greatest communication tool.
This always felt like a giant ADA lawsuit waiting to happen.
Captchas are primarily to punish users for not allowing tracking, or using the “right” services, they may prevent some bots as a side effect (or a pretence from the provider) but it’s mostly for google and cloudflare to abuse their monopolies.
Cider9986
today at 6:43 PM
Google I would say yes, but what does Cloudflare gain? They don't run an ad network. Generally I'd say Cloudflare is pretty good to have as a guardian of the web compared to other options.
They protect free speech and allow Tor users. Ever tried completing a reCaptcha on Tor?
Cloudflare gains things like this:
https://blog.cloudflare.com/introducing-pay-per-crawl/
https://developers.cloudflare.com/browser-run/quick-actions/...
They create a new problem and sell the solution.
davidfischer
today at 8:31 PM
Nowadays, somebody can just ask claude to build them a scraper/bot that hooks into a proxy network and all of a sudden they can easily send 20k+ reqs/min from hundreds or thousands of IPs cycling them as they get rate limited or banned. In my work, the scrapers have gotten way more aggressive in the last 2 years or so. Frankly, I'm happy there is a solution.
There may be things to criticize Cloudflare for, but the problem of bots and scrapers destroying the open web was getting worse no matter what.
YeahThisIsMe
today at 8:16 PM
God damn it.
Tin hat folk say Cloudflare is CIA. I dunno
This feels like the kind of thing where, "you must be at least this human to pass" and that it just otherwise mostly wastes your time if you're a robot would cover most of what Captchas are useful for.
Like, if it takes you 3-5 seconds to get through a captcha as a human, as long as every single event has that effort added, the impact to something trying to use/reuse the end-page is way worse if you're a robot than if you're a human.
I can see a few usecases where it would still be valuable to continue the game of cat-and-mouse, but I feel like solving for consistency of human experience of your website, may actually be more punishing to anything trying to bypass it.
Isn't this solveable by Anubis or similar? if you just want to add some costs to bots you can do that directly and it'll be pretty invisible to humans
edelbitter
today at 6:50 PM
But.. the task was never "detect this" but always "detect this within acceptable constraints".
Sure, once you collect enough bits, you can tell that its me. And if you know from other sources that I am human, that solves your immediate problem.
But if you do that, you have still failed at the task of detecting certain kind of abusive behavior without harming my anonymity.
CarbonCycles
today at 6:41 PM
Appreciate this article...shows some interesting insights on how humans "behave" vs agents.
technotarek
today at 5:32 PM
Apparently CloudFlare’s turnstile can’t, as evidenced by several public-facing CRUD and mail routines we maintain that no longer are warding off the spam.
Yeah, we benchmarked against a few bot detection provides end of last year (https://research.roundtable.ai/bot-benchmarking/), and Turnstile didn't do great when it came to AI agent detection. We hypothesized that Turnstile primarily focuses on device/network characteristics, which AI agents can bypass
Meanwhile the moment I (a human, of which I'm reasonably confident) see a Cloudflare captcha I nope immediately out of the site and block it forevermore in Kagi. It's not worth the waiting game. "Verifying..." lasts ages.
The anime girl captcha works fine and provides no such annoyance.
You seem to think that having a random anime girl is not an annoyance. anything that deviates from showing me the content that I've requested is an annoyance. Just because you prefer A over B does not mean that A is not still an annoyance.
docheinestages
today at 5:16 PM
I think it's just a game of cat and mouse. It might be easier to catch naive AI agents that are not fine-tuned for specific CAPTCHA tasks with human behavior, can't recognize new challenges, don't know when to stop and ask a human, and just want to brute force their way with limited or no specialized harness and tools available.
This is relatively close to our conclusion from the paper: unless agents are specifically trained for the task and know all the information ahead of time, they're not able to generalize from one cognitive CAPTCHA to another
- LLMs can't learn, therefore, LLMs are only good for things on which they are trained.
- Captchas are not friendly with trial and error, so agentic solutions also don't help.
- It's impractical to train LLMs on everything.
- We humans are capable of creating infinite ways of captchas.
While each of these sentences is true, captchas will always win against LLMs.
A captcha a LLM can't be trained to defeat is likely a captcha humans will struggle quite a bit with.
PeterStuer
today at 6:38 PM
So now I have to fail the capcha to prove I'm human, but in the right way? We don't hate these people enough.
Adversaries do not have to wait for LLM models to evolve to mimic human process, they can simply evade the detection JavaScript that evaluates similarity. JavaScript is visible, can easily be reverse-engineered.
I don't think I've ever known of a captcha that handles the actual result decision in the front end. It's universally just the javascript required for some fancy puzzle UI, which forwards the state to some other endpoint to determine where you're redirected to (CF turnstile) or what signed token should be included in the form request (reCAPTCHA)
I should have been clearer and specific: state management is done on the backend, but collecting behavioral biometrics and device fingerprint is done using JavaScript, which can be manipulated.
I’ve been using Claude Opus 4.7 with Chrome MCP, and it has worked successfully about 95% of the time. However, I’ve failed various hCaptcha challenges.
amirhirsch
today at 5:57 PM
The thing many people miss is that the challenge itself isn't the primary signal. The challenge creates an opportunity to observe user activity. You're browser is also fingerprinted.
What happened to adversarial attacks? I.e. noise that makes an image look like something else to a classifier than to humans. I guess frontier LLMs are no longer vulnerable to those?
BiteCode_dev
today at 5:03 PM
Until they learn to do that. So cat and mouse. So nothing new.
They already have. Claude and OpenAI are not trying to write captcha-defying AI agents. These tests wouldn't hold up as well against proper bot operators who mimic user behaviour. However, the signals are still valid as part of a larger toolset.
think the point is that they can't just "learn to do that", because to do so would mean solving human mind (that famously hasn't been going well)
sigbottle
today at 5:15 PM
Well no, the idea is a tradeoff between interfaces and telemetry.
OK, the agents don't click in the same way as humans. You learn that, what about mouse hovering telemetry, time spent, etc. And one of the most extreme is to force biometrics - a lot of telemetry, breaks the interface a lot - but hey, you have assurance.
And none of these tradeoffs require understanding the deep processes of the human mind. Just, map is not the territory, how you do game the map harder and harder and how do the mapmakers respond to that?
did you look at the paper? they specifically look at mini tasks with cognitive processes (Eg what dictates the strategy of how people solve tasks)
CamperBob2
today at 5:43 PM
LLMs can solve original math problems at the IMO level and beyond, and you might be talking to one now. I don't think they are going to have problems with any CAPTCHA short of separate device attestation.
Whatever mechanism the paper proposes, rest assured it can be trained on.
dpoloncsak
today at 5:45 PM
until Google trains an AI model off that data, too