Cloudflare is known to use fingerprinting to detect scrapers For example, they use JA3 fingerprints and match them against the UA to block stuff like cURL while allowing OkHttp (Android clients) - but this can be easily be spoofed with packages such as CycleTLS [1].

I don't want to defend them, because they gate away a good chunk of the internet with their "bot protection", but unless you do PoW (which is also ecologically a nightmare), probably fingerprinting is the way to go - completely destroying the privacy of everyone involved.

Cromite, a privacy conscious fork of Chromium for Android, has constantly issues with CloudFlare Turnstile [2] because they (Cloudflare) try to fingerprint it in multiple ways in order to pass the challenge. The only way to get it to work would be to join the CloudFlare Browser Developer program - which requires signing an NDA. Rightfully so, the project maintainer didn't want to do it.

If you want to see the extent of what CloudFlare does to fingerprint the browsers, just have a look in the issue [2] and see which flags need to be disabled in order to allow CloudFlare to pass the challenge.

I understand both sides, but at least CloudFlare could be flexible enough to fall back to PoW instead of just blocking people from sending forms or accessing websites...

[1]: https://github.com/Danny-Dasilva/CycleTLS

[2]: https://github.com/uazo/cromite/issues/2365

• jwr

  today at 4:40 PM

  > I don't want to defend them, because they gate away a good chunk of the internet with their "bot protection"

  They also gate away a good many people with their "bot protection". I am extremely worried about how so many seem to have outsourced the control over who can access their websites to a company, with no second thoughts whatsoever.

  • binaryturtle

    today at 5:55 PM

    I can no longer access any website that's "protected" by Cloudflare. As soon a website enables that stuff… "Shoot, another one bites the dust." I wonder if the website owners realise at all how many actual users they lose by this sort of "protection."

  • denysvitali

    today at 4:58 PM

    They sometimes have to comply with legal requests (which I understand), but at the same time they have a huge market share - which means that the internet is becoming less and less decentralized and more in their control. We've seen the effects of that in previous outages...

  • stackghost

    today at 5:22 PM

    >I am extremely worried about how so many seem to have outsourced the control over who can access their websites to a company, with no second thoughts whatsoever.

    I think the Web is on its last legs, anyway. Generative AI and LLM-instead-of-search has destroyed what little value remained.

• sandeepkd

  today at 5:29 PM

  > I don't want to defend them, because they gate away a good chunk of the internet with their "bot protection", but unless you do PoW (which is also ecologically a nightmare), probably fingerprinting is the way to go - completely destroying the privacy of everyone involved.

  Bot protection with fingerprinting is just an illusion. Any signals like this which is on client side can be spoofed by an above average person. Fingerprinting is just way to consolidate the market for advertising business. Assigning Reputation to residential IP addresses and commercial blocks is is another approach to achieve the desired result. Providers would be a lot more careful to allow their IP addresses for misuses, however turns out that it would bring down the DDOS business on both sides, attackers and protectors.

  Ironically, more than often its the same companies that invest in building their own bots and finding ways to stop bots from other companies.

• b65e8bee43c2ed0

  today at 3:47 PM

  it's all for nothing, because Cloudflare's scraping protection works about as well as a $5 padlock - good enough to dissuade bored teens, not good enough to dissuade even an amateur burglar. if someone wants to scrap your publicly visible data, they will. there's nothing you can do.

  • ACCount37

    today at 3:49 PM

    At the same time: it sure works well enough to annoy anyone with a "bad ASN" IP with 80 captchas a day.

    • shideneyu

      today at 4:18 PM

      exactly that's what I was thinking... like the day they provided a solution to the issue they posed

  • mootothemax

    today at 5:28 PM

    Exactly. I’m constantly amazed at how little you actually need to bypass CF, Amazon, Azure WAFs and so on (Incapsula springs to mind too). When you look at the code you’ve come up with, it’s actually quite small and compact.

    More to the point, these systems actually help scraping because proof of work unlocks essentially unlimited scraping, in my experience.

    That said - from my experience on the other side, sure you can’t stop people like me or you, but you can stop 99% of the others. That’s more than worth it operationally.

• petu

  today at 4:25 PM

  > but unless you do PoW (which is also ecologically a nightmare)

  Can you expand? I don't see a problem with some napkin math. 5W load for 2 seconds is 0.002Wh (we have to let smartphones pass and not by doing PoW for 10s of seconds). 8 billion checks a day for a year = 8GWh.

  • denysvitali

    today at 4:55 PM

    I stand corrected. It's not a nightmare scenario (as for Bitcoins) - but I'm still of the idea that "useless" computations should be avoided (as we should avoid having 10MB websites).

    In any case, according to some napkin math done by Kimi 2.6 (which by itself is probably already consuming more than all of my PoW challenges for the upcoming 5 years) - the situation looks incredibly in favor of PoW: https://www.kimi.com/share/19e7ef40-a432-8912-8000-0000b4a71...

    Which makes me wonder why CloudFlare isn't switching to this already

    • dcrazy

      today at 5:03 PM

      Because it doesn’t solve the problem of residential botnets.

• PearlRiver

  today at 3:20 PM

  This is why I have two separate browsers. If you want to do official stuff like paying for things you need to get through cloudflare.

  • notafox

    today at 4:16 PM

    You can use Firefox with different profiles and configure it to launch particular profile directly, without launching default profile and using about:profiles.

    Firefox with a non-default profile can be created like that:

      ./firefox -CreateProfile "profile-name /home/user/.mozilla/firefox/profile-dir/"
      # For, say, cloudflare that would be:
      ./firefox -CreateProfile "cloudflare /home/user/.mozilla/firefox/cloudflare/"
    

    And you can launch it like that:

      ./firefox -profile "/home/user/.mozilla/firefox/profile-dir/"
      # For cloudflare that would be:
      ./firefox -profile "/home/user/.mozilla/firefox/cloudflare/"
    

    So, given that /usr/bin/firefox is just a shell script, you can

        - create a copy of it, say, /usr/bin/firefox-cloudflare
        - adjust the relevant line, adding the -profile argument
    

    If you use an icon to run firefox (say, /usr/share/applications/firefox.desktop), you'll need to do copy/adjust line for the icon.

    Of course, "./firefox" from examples above should be replaced with the actual path to executable. For default installation of Firefox the path would be in /usr/bin/firefox script.

    So, you can have a separate profiles for something sensitive/invasive (linkedin, cloudflare, shops, banks, etc.) and then you can have a separate profile for everything else.

    And each profile can have its own set of extensions.

    • t_mahmood

      today at 5:29 PM

      You do now do this from `Profiles` menu too, without going down to CLI path. It's extremely simple now.

    • ferfumarma

      today at 4:46 PM

      Except that fingerprinting means that both profiles are actually tied together by cloudflare (and other tech companies)

      • VoidWhisperer

        today at 5:55 PM

        I think the idea is that they have the functionality that cloudflare is using to generate the fingerprint (like webGL in this case) disabled in their non-cloudflare profile and only use the cloudflare profile to do things they have to that are behind cloudflare

  • helterskelter

    today at 3:40 PM

    Firefox added profile switching recently. Works good.

    (That said, I still keep separate machines. One for doing "official" things, the other for everything else)

    • notafox

      today at 4:18 PM

      > Firefox added profile switching recently.

      I think this was as recent as 25 years ago?

      Recently they added some new UI. There was and still is (I think) classic Profile Manager UI, which you can launch with

        ./firefox -ProfileManager
      

      or access UI in about:profiles.

      But you don't have to use any of those anyway - see my comment above (a response to parent).

      • opem

        today at 5:01 PM

        They actually have at least 3 kinds of profile: 1. containers - As they say its somekind of sandbox, technically a profile 2. profiles that are accesible through about:proflies, which they had for years, and probably the one you are talking about... 3. New profiles that comes with a pop-up much like how chromium browsers shows it

      • thayne

        today at 5:35 PM

        The old UI was pretty difficult to use, and hard to discover unless you knew where to look though.

    • ajb

      today at 3:57 PM

      Odd - they've had that for years, but only on the command line. Wonder if it's different under the hood? They also have firefox containers which also never quite became a first-class feature (you have to install a plugin).

      • today at 4:21 PM

    • b65e8bee43c2ed0

      today at 3:53 PM

      >Works good.

      does it? same binary, same machine, same display, same 781 other heuristics.

> Plus privacy.resistfingerprinting isn't enabled even when selecting "Strict" "Enhanced Privacy Protection" in the settings, great job there Mozilla.

For good reason. I've run that setting for ages but I kept having to disable it and add workarounds because websites would break in weird ways. Timezones in scheduling websites being messed up nearly made me miss a couple of appointments. There's no way to tell the user Firefox isn't broken without displaying a permanent banner like "if websites are broken in any way or you see weird glitches or your computer's time is wrong or fonts look weird or videos don't always work right, click here to disable fingerprinting protection".

Interestingly, Turnstile breaks with resistfingerprinting but works with fingerprintingProtection, I guess the latter takes this crap into account.

• croes

  today at 5:22 PM

  Maybe a good reason for not enabling it by default but a bad reason to not enabling it for strict settings.

  I somewhat expect breaking sites with strict settings, I don’t expect an still wide open tracking path.

  That’s deceiving.

Is there a deal between Google and Cloudflare to make non-Chrome browsers harder to use? The pressure to use Chrome keeps increasing, and the amount of ad filtering you can do in Chrome keeps decreasing.

Thanks, i did not know about `privacy.resistfingerprinting`

I'll make sure to fail all cloudflare turnshit in the future.

• gruez

  today at 3:04 PM

  I have it enabled and turnstile works fine.

  • jeroenhd

    today at 5:04 PM

    It breaks Turnstile for me on Android. Had to restart the browser for it to take effect of course.

So if you need to prevent bot abuse, but also don't want an ugly captcha every time someone goes to sign up, is there a better option?

• ribtoks

  today at 3:27 PM

  Use proof-of-work captchas, many are private by default. Look into Private Captcha or Cap captcha.

  • mootothemax

    today at 5:33 PM

    Speaking from the scraper’s perspective, I like proof of work; a ten year old 96-core server will cost a couple of quid to run for a few hours and will grab an absurd number of pages thanks to the access granted by repeatedly solving proofs of work. Small slick codebases too!

  • phoronixrly

    today at 3:43 PM

    How does proof of work stop bots?

    • stephantul

      today at 3:46 PM

      Because it destroys the economics of scraping. It’s too expensive with proof of work, or at least not as economically viable

      • gruez

        today at 3:52 PM

        Depends on what type of scraping you're trying to stop. For the dumb scrapers that would try to scrape every page on a git forge (for which there are a bazillion pages for a modest project, because of how the site works), yeah it might deter them enough to stop. For anything high value (eg. reddit comments or retail prices), 10s of cpu time isn't going to stop them.

        • thayne

          today at 5:44 PM

          If it's high value, there isn't really much you can do that will be completely effective. Traditional captchas can often be beaten by AI, or by "captcha farms" where impoverished people are paid pennies to complete captchas. Fingerprinting can be beaten by using a full browser to make the requests. Basically anything you do is just a matter of making it more expensive for bots to access it.

        • stephantul

          today at 5:03 PM

          Sure, the whole premise is exactly that proof of work reduces the value of scraping, while having negligible impact on users. If the data is so valuable that bot operators are willing to pay 10s of cpu, then other measures are necessary.

          Nevertheless even for these high value cases, you can still argue that it disincentivizes the business model, it becomes less efficient.

        • pmontra

          today at 3:59 PM

          It will not scare away bots but 10 seconds of wait (CPU or only a sleep) will turn away many real users. "This site is so slow, I'll use something else." A kind of reverse captcha.

          • Hnrobert42

            today at 4:33 PM

            Maybe, the proof of work can run in the background.

            • btown

              today at 5:28 PM

              Or it can run as part of a checkout wizard's "verifying your browser and processing your payment, don't close your tab" step.

          • today at 4:08 PM

    • ray_v

      today at 3:50 PM

      If it gets too expensive/time-consuming to scrape then it won't happen at scale (as much)?

• ImPostingOnHN

  today at 3:27 PM

  The tool "Anubis" uses proof of work instead

  • BetterThanSober

    today at 3:50 PM

    With a tuned cool down period this isn't a problem, especially if you frequent the sites. OpenWRT uses Anubis and usually when I need to peruse their site I'm on a very low-end device. I prefer waiting much more over finding Waldos

    But in principle I agree that there's no good answer to this, scraping _is_ useful and I bet most of us here had scraped something, it is AI company and their use of human's material for training without consent and return that led us to this (I know botting exists in forum since forum is a thing but it is easily solved by human moderators and keyword filter)

  • timpera

    today at 3:31 PM

    Anubis often takes more than 60 seconds to complete on low-end devices (especially old smartphones). It seems like there's no good solution.

    • QuantumNomad_

      today at 4:18 PM

      But after you’ve completed the Anubis PoW challenge for a site, it remains valid for some amount of time.

      So it’s not quite as horrible as it sounds.

      I have setting up Anubis for my own sites on my todo list. And I wish more people did it too. I don’t really mind waiting a little bit extra every now and then before the page loads. What I do mind is ReCaptcha asking me to click all the pictures with buses in them etc. And especially when I have to do it several times over before it’s happy. I’d rather wait a minute for a page to load than to ever solve a ReCaptcha again, if given the choice.

    • dangus

      today at 3:40 PM

      That must be really low end then. I’ve never seen it complete in a timeframe that was slower than “I can’t even read the page before it redirects”

      • titularcomment

        today at 5:18 PM

        My guess is its an implementation error, not an hardware limitation. I have two 10-year-old devices and one passes instantaneously while the other halts for a good half minute every time.

    • ImPostingOnHN

      today at 3:36 PM

      There's not an easy, perfect solution, for sure. Newer phones get faster, but spammer compute gets cheaper.

      Some sort of decentralized trust web seems like another option, though less viable.

      • WesolyKubeczek

        today at 3:54 PM

        One of unexpected outcomes from AI-induced hardware shortage may be that, in fact, compute won’t be getting cheaper and may in fact get more expensive…

  • phoronixrly

    today at 3:44 PM

    How does Anubis stop bots?

    • redwall_hp

      today at 5:49 PM

      Anubis is designed to stop a certain class of badly behaved bots. It intentionally doesn't run if a bot identifies itself with a UA, such as Googlebot, because then you can rate limit it or block by UA and with other tools.

      Anubis is active when a user agent looks like a web browser (e.g. contains the "Mozilla" substring every major browser uses). The reverse proxy serves an interstitial page that does a proof-of-work check, validated server side, setting a cookie if it passes.

      This means a legitimate user won't constantly get the proof of work check, because they already passed it. But AI bots rotating through tons of residential IPs to scrape your forum or git forge or whatever will be slowed down.

      Overall, I like the idea. It's unobtrusive, privacy preserving, and seems to be working out well for a lot of sites.

    • basilikum

      today at 5:08 PM

      The real answer is that it makes sites behave different requiring the bots to make slight adjustments.

      And there are just not enough sites using Anubis for the people and companies running the bots to care to do that.

      If you do care bypassing Anubis is trivial.

    • xena

      today at 3:54 PM

      Bots don't execute JavaScript or follow complicated redirects.

      • pwg

        today at 4:08 PM

        Bots don't [currently] execute JavaScript or follow complicated redirects.

        They don't now, but enough "high value to the bots" pages turning on JS or complicated redirects will simply result in the bot authors adding JS execution or redirect following so they can continue "botting" the sites they want to scrape.

        It's a hole with no bottom. Each one-up on the anti-bot side will eventually be handled on the bot side.

> Plus privacy.resistfingerprinting isn't enabled even when selecting "Strict" "Enhanced Privacy Protection" in the settings, great job there Mozilla.

That pref is there for the Tor Browser.

• konform

  today at 5:38 PM

  It's enabled by default in Tor Browser and I'm not sure it can even be disabled?

  Also enabled by default for Konform Browser and Mullvad Browser, which borrow many of the privacy- and security-related patches from Tor Browser.

I tested this extension that I've been using for a long time on the turnstile page and it got through, fwiw. I think it's a bit more subtle than how resistfingerprinting works but not sure what the privacy tradeoff is.

https://github.com/kkapsner/CanvasBlocker

• tosti

  today at 5:42 PM

  Looks cool. And I wonder why I'd run this over JSshelter. It appears to do the same thing, no?

• BoingBoomTschak

  today at 5:56 PM

  Thanks for the report, I've been running this for a long time.

In other words, Cloudflare requires you to substantially increase your browser’s attack surface in order to visit websites.

Doesn't this mean we just need to make the webgl fingerprint resistance implementation smarter? Instead of explicitly rejecting webgl access or responding with dummy data, respond with data that is random within space of N common and reproducible patterns. E.g. emulate webgl implementation of some low spec but actually popular devices.

• btown

  today at 5:38 PM

  The last screenshot in the OP article mentions that "a browser extension... adding random noise to canvas data" can be detected. Which isn't to say this perfectly detects all such randomization, but it's certainly an active part of the arms race.

• bflesch

  today at 3:54 PM

  All of those advanced features should be enabled on a per-website basis but unfortunately even browsers whose marketing focuses on privacy don't allow you to do that. Same with TLS root CA certificates, there is no way to configure that a certain CA can only create certificates for certain domains.

I did warmups in Grub Crawler to fight this: https://deepbluedynamics.com/grub

They use all kinds of obscure APIs, which you'll learn if you're privacy/security conscious and disable random web APIs that are of no use to YOU as a web user, but only can ever serve the people who serve you stuff or want to hack you or track you.

Normally websites feature test and just skip using obscure disabled APIs, or more likely, websites don't use those APIs at all or only tracking scripts use it, which are already optional usually.

Problem with CF is that if you want increased security they'll prevent you from gaining it everywhere, even on sites they don't protect, or prevent you from accessing services even the ones you paid for. Browsers don't allow disabling APIs per domain, so you're either at risk everywhere or you're blocked from accessing a lot of things for no particular reason.

CF can't be bothered to feature test.

This company makes the internet unusable if you value privacy and use VPNs or whatever. Evil.

This blog post is filled with false assumptions.

>Turns out it's because Cloudflare wants to have a fingerprint of your device via WebGL, the only reason for doing this would be tracking.

> So Cloudflare just banned all WebKitGTK browsers as I guess they put an exception for Safari.

This is false. I ran firefox with:

* hardware acceleration disabled (so software renderer, nothing to fingerprint)

* resistfingerprinting enabled, including letterboxing with default window size

* webgl disabled

* VPN enabled

* In a Windows VM

By all accounts this should be the most suspicious fingerprint ever, but turnstile happily lets me through. If they want to track people, they're doing a pretty bad job. My guess is that OP's browser is getting banned because his WebKitGTK has a weird fingerprint, not because of webgl or whatever.

> Such things are blocked in WebKit, and have been for years. Meaning it's tracking so awful that even Apple would block it, and as far as I can tell it's not the kind of privacy protection you can easily disable in it.

This is also false. Webgl fingerprinting works just fine on Safari. They might try to mitigate it by adding some noise, but that's not so different than what firefox does, and is certainly not "blocked".

• konform

  today at 5:43 PM

  I think you're making plenty assumptions yourself..

  Official Firefox can be leaky unless you build it yourself with some build-time changes or use a fork with such[0]. Am I guessing right that you still have Webcompat, RemoteSettings, and Nimbus enabled still? How do you know a compatibility intervention isn't causing your browser to open the kimono just enough to "unbreak the page"?

  > My guess is that OP's browser is getting banned because his WebKitGTK has a weird fingerprint, not because of webgl or whatever.

  My guess is a different flavor of the same: Not matching an expected fingerprint (simplified: whitelist vs blacklist approach) combined with other factors.

  [0]: I'm currently aware of Tor Browser, Konform Browser (am dev), Mullvad Browser, and to a certain extent Waterfox, LibreWolf, and r3df0x doing this.

• jeroenhd

  today at 5:05 PM

  Enabling resistfingerprinting on my Android phone shows me the same error screen. It's not just webkit.

  fingerprintingProtection works fine on the other hand, but then again that's intentionally less intrusive.

• shiomiru

  today at 3:32 PM

  > My guess is that OP's browser is getting banned because his WebKitGTK has a weird fingerprint, not because of webgl or whatever.

  So why is Cloudflare saying the author got blocked because of WebGL?

  > > Such things are blocked in WebKit, and have been for years. Meaning it's tracking so awful that even Apple would block it, and as far as I can tell it's not the kind of privacy protection you can easily disable in it.

  > This is also false. Webgl fingerprinting works just fine on Safari. They might try to mitigate it by adding some noise, but that's not so different than what firefox does, and is certainly not "blocked".

  While I don't have an iDevice to try, the assumption that they are special cased is fair... because they are: https://blog.cloudflare.com/eliminating-captchas-on-iphones-...

  (Yes, this is basically WEI in a shinier package.)

  • gruez

    today at 3:37 PM

    >So why is Cloudflare saying the author got blocked because of WebGL?

    No idea. I can't even reproduce the error OP got with webgl disabled.

    https://litter.catbox.moe/y42l22k97tgv96nx.png

• superkuh

  today at 3:31 PM

  Yep. Cloudflare and cloudflare's customers don't care about blocking people that use non-standard browsers (or accessible browsers, or feed readers, or whatever). Using cloudflare defaults is basically saying, "Only major corporate browsers released in the last year or two can access this site."

Adding noise to a canvas element is a mistake anyway. It means you can't develop a proper paint program using web technologies because your browser will mess with the image.

• tosti

  today at 4:54 PM

  You can still do that, but it may not be rendered correctly in a screenshot.

"This makes your browser appear suspicious because it looks like you're trying to hide your identity."

Yeah, this needs to be burned to the ground.

• gruez

  today at 3:46 PM

  Bad optics aside, it doesn't actually reflect reality. See my other comment. You can enable basically all the privacy settings and still pass turnstile. Tor browser in a VM passes it, of all things.

  https://litter.catbox.moe/gaizpk692bhhs6b7.png

  • JoshTriplett

    today at 4:04 PM

    Any idea what the difference is between your setup and the one in the article that failed with fingerprint-resistance enabled?

    • gruez

      today at 4:08 PM

      He's using a custom browser, apparently: https://hacktivis.me/projects/badwolf

      • JoshTriplett

        today at 4:24 PM

        I'm talking about the screenshot from Firefox.

        • gruez

          today at 4:40 PM

          It didn't fail for him in firefox, even with privacy settings enabled.

          • JoshTriplett

            today at 5:22 PM

            It tripped "Canvas Randomization Detected". See the last screenshot.

            Cloudflare's demo page still treats that as a pass, but complains about it. As is often the case with Cloudflare, I expect that they'll then take no responsibility for sites that use more aggressive settings.

Dont like it but is a reality due to bots

As turnstile users on several of our sites, I think we need to revisit that decision.

• sammy2255

  today at 2:54 PM

  Out of curiosity, why did you have it on in the first place?

  • nulledy

    today at 4:11 PM

    Bot rejection for contact forms. Better UX than reCaptcha.

...in the age of AI, does anyone have an actual solution for keeping out bots while preserving the privacy of humans?

Obviously this is terrible, but I think there's a possibility it's the least terrible option? Another option is IP reputation, which I think is worse. Or scanning a code with a non-rooted phone, which I think is even worse than that!

• fidotron

  today at 2:57 PM

  > ...in the age of AI, does anyone have an actual solution for keeping out bots while preserving the privacy of humans?

  There isn't one, and pretending otherwise is nonsense because humans will always provide their credentials to something to act on their behalf.

  In the limit you end up with Chinese phone farms.

  • tardedmeme

    today at 3:06 PM

    Right. Botnet operators love cloudflare because they make so much money renting out compromised machines to pass their tests.

• thisislife2

  today at 3:21 PM

  The only solution is regulation. If all content created by anyone has a copyright, how does an implicit opt-in (which is what happens if you don't create a robots.txt file for your website) for scraping make any sense? Moreover, even if you have a robots.txt, AI (or whatever) bots often don't respect it (or use workarounds - they outsource scraping of such "restricted" sites to unethical third-parties to get the data; Meta has even resorted to piracy, openly!). So clearly, the logic and the "honour system" has failed.

  Cloudflare, Google Captcha, HCaptcha etc. are all shitty technical solutions because, as we are all discovering, it comes at the cost of our privacy (i.e. our personal data may monetise these services) and / or our computing resource and time. If current copyright laws aren't sufficient to prevent this, we have to acknowledge the system is broken. The answer could be enhancing it with some kind of Digital Millennium Copyright Act (DMCA) -like laws, but in favour of the creators against BigTech or rogue actors.

  - Web-scraping and copyright law - https://www.neudata.co/blog/web-scraping-and-copyright-law

  - Why DMCA Claims Against Web Scrapers Face Long Odds - https://capstonedc.com/insights/why-dmca-claims-against-web-...

  • oceanplexian

    today at 3:48 PM

    Or you could let information be free, at least the stuff that’s on the public net.

    As for issues like bots overloading websites or using too many resources scaling laws will take care of it quickly, it’s not like you can’t serve thousands of RPS from a Raspberry Pi these days.

  • ImPostingOnHN

    today at 3:29 PM

    I don't think regulation will stop web scraping, not least of which because it can be done from locations outside the jurisdiction of the regulations.

    > we have to acknowledge the system is broken

    The system is broken. It probably takes, what, 10 seconds or less to use a residential or foreign proxy, 6+ months to internationally track and prosecute a single offender? So like a million times more effort going the regulatory route.

    • thisislife2

      today at 3:36 PM

      Just as criminal laws don't end all crimes, copyright laws and anti-scraping regulation won't end all scraping. But it will greatly reduce it and limit it to rogue actors. Two examples I can cite here are the laws against email spams and laws against unsolicited marketing calls - they had a definite impact in reducing both (even in India, from where I am, where implementation of laws are often lax).

      • JoshTriplett

        today at 3:40 PM

        Exactly. Bot activity is a problem of volume, not all-or-nothing. Solving 95% of it would be a win.

• jeroenhd

  today at 5:07 PM

  Remote attestation should still be possible with a rooted phone if phone manufacturers weren't so shit. If the attestation happens at hardware level, it doesn't matter what programs or kernels you're running.

• ravenstine

  today at 5:20 PM

  Or maybe we can actually start paying for all the things we use on the Web, making it prohibitively expensive to deploy fleets of bots.

• cr125rider

  today at 3:05 PM

  And identifying a bot that is acting on my behalf. Claude go search this topic is basically the same as Googling something and clicking on the results. Human driven AI searching needs to be in a different box than AI scraping for training data.

  Which sounds extremely difficult to differentiate

  • JoshTriplett

    today at 3:32 PM

    Hopefully it stays that way; "a bot acting on my behalf" is still a bot. At least it's often a well-behaved bot and uses a user-agent that can be detected and blocked.

• Gander5739

  today at 3:19 PM

  You don't need a non-rooted phone to pass captcha checks, I have a rooted phone and can pass the captchas that ask you to scan a qr code. But I doubt phones without google services would manage.

  • HWR_14

    today at 5:44 PM

    How does scanning a QR code prove any kind of captcha?

    • Gander5739

      today at 5:50 PM

      https://support.google.com/recaptcha/answer/16609652 - it just launches the verification service.

• spacedoutman

  today at 3:13 PM

  Private invite only internets

• csomar

  today at 3:08 PM

  They are not a problem unless you "believe" it is a problem. I estimate around 20-25K hits to my website from bots per day and I have all cloudflare protections disabled. Any decently optimized server should be able to easily handle that. (it's roughly 1 request every 3 seconds).

  • specialp

    today at 3:22 PM

    Yes and that is just the bot background radiation of the internet. I run a primary source of information site and these botnets are aggressive to a DDOS level. All to do some sort of scraping. Because they have sophisticated enough tactics to DDOS us if they wanted to. However I am not sure their objective as they have wasted enough of our resources to have scraped all our content 1000s of times over. That 25k traffic is a couple of minutes for us. And that adds up. 80-90pct of our traffic is this

  • HWR_14

    today at 5:45 PM

    Assuming that the bots aren't repackaging your content and preventing users from seeing your blog by serving that content to them first.

  • thisislife2

    today at 3:24 PM

    True. But it still wastes your server resources, right? And it's sad that you have to accept that as part of the "cost" of hosting a site ...

    • ndriscoll

      today at 3:46 PM

      What resources are you concerned about? An n100 minipc should be capable of serving something like a blog at 20k+ requests/second (or saturating its network).

      • today at 4:56 PM

• doctorpangloss

  today at 3:08 PM

  web environment integrity

• malka1986

  today at 2:58 PM

  > keeping out bot

  You can forget about it. It is not possible. Simple as that.

  • Wowfunhappy

    today at 3:01 PM

    Let's say I'm selling concert tickets. How do I prevent bots from buying up all the tickets and scalping them?

    • ranguna

      today at 4:18 PM

      Do it like plane tickets do, tie a ticket to an identity + buyback up to a week or so before the concert in case someone wants to cancel (or authorize the transfer and capture only a week before). Ask for ID and ticket at the entrance.

    • ndriscoll

      today at 3:52 PM

      Sell them via a Dutch auction. Eliminate the arbitrage opportunity for scalpers and make more money in the process.

      • dcrazy

        today at 5:06 PM

        That’s how you wind up with only kids of millionaires at your Taylor Swift concert.

        • queenkjuul

          today at 5:59 PM

          So a Taylor Swift concert

    • MyMemoryfails

      today at 3:12 PM

      I'd simply check filling speed, even with browser's autocomplete humans are slow due needing click submit.

      Then when it's "processing", do them in bulk and prioritize slower users. There's huge opportunity do bot checks after checkout without affecting user experience.

      Also on product launches you could add unique field which requires user to input, for example that way bots can't prepare for launches.

      • fragmede

        today at 3:39 PM

        huh. no wonder my password manager's auto submit triggers bot detection (it's a fairly popular one).

    • luckylion

      today at 3:03 PM

      Tie them to the buyer's identity, offer at-value buy-backs until X weeks before event, disallow resale.

• ashishbijlani15

  today at 5:36 PM

  [dead]

Say no to malware - say no to Cloudflare

Firefox has so much built-in tracking it seems they want to push me to build my own browser. For example every time you open the settings there are several ways they are sending out pings to certain extensions.

Also by default addons.mozilla.org is a privileged site so of course they include google tracking in it and they get the proper fingerprint no matter what you have configured.

• konform

  today at 5:53 PM

  If you are this motivated (I am!), how about joining forces on Konform Browser? Radio silence and remote third-party integrations disabled by default and generally sane and conservative defaults respecting old-fashioned notions like individual consent and data-protection regulations.

  Aside from general dev, could use a hand in bringing it to more platforms (mobile and flatpak are frequently asked) and taking a closer look at fingerprinting protections and what's currently tripping up the turnstile.

  https://codeberg.org/konform-browser/source

I wondered about that too. So they allege that bots require that everyone now has to ID to the big service providers. Very dystopian situation. Skynet is currently winning the war.

Please, anyone from EU (US is doomed rofl) create a petition to ban browser-fingerprinting in EU, across all existing browsers.

I'm not good at creating petitions but can happily sign it. Also with stop killing games and anti-chat control.

I can imagine this can get a traction, if it's explained in youtube video to "normal" people.

• fidotron

  today at 2:59 PM

  A better solution would be to make webgl, webgpu and (especially) webrtc have some sort of prompt before they can be in any way used in that fashion, but this will absolutely destroy web ux Windows Vista style.

  • JoshTriplett

    today at 3:38 PM

    And then the gatekeepers like Cloudflare will say "please hit accept in order to verify your browser and access this site".

  • richwater

    today at 3:09 PM

    You mean the "Accept Cookies" banner that has become a complete joke? Pass

    • fsflover

      today at 5:58 PM

      This is actually illegal under GDPR.

    • MyMemoryfails

      today at 3:23 PM

      I think he means browser permissions, for example when browsers want notify or record your mic theres a permission check something similar for webgl.

      • J-Kuhn

        today at 3:42 PM

        Fun Fact: When Cookies were introduced into Netscape, you got a browser permission prompt. Then browser vendors set it to allow by default.

        And then legislation required those consent boxes back, so everyone built their own, instead of demanding that the default should be changed back.

    • bflesch

      today at 3:57 PM

      It's about explicitly deciding to allow certain capabilities on a per-website basis. No major browser allows defense-in-depth via fine-grained website permissions.

      Even simply changing the user agent was sabotaged at Firefox, and choosing one user agent per domain is wishful thinking.

• jeroenhd

  today at 5:10 PM

  Fingerprinting is just an implementation, banning it will just drive these companies to invent new tricks. That's why the GDPR doesn't specify any technical tracking methods, whether you're using cookies or fingerprinting or a camera drone looking at the user's screen, tracking without consent or good reason is banned.

  I doubt politicians care much about fingerprinting, though. They're more afraid of actual businesses getting attacked by bots than they are about Linux users with weird setups not being able to access some websites.

• koolala

  today at 2:55 PM

  a. Accept All

  b. Accept Only Necessary Fingerprinting

[dead]

What? Big tech company is evil? No way! I thought cloudflare were good guys...

• aleksandrm

  today at 2:55 PM

  What gave you the impression that Cloudflare were the good guys?

  • tardedmeme

    today at 3:06 PM

    Probably everyone on HN singing their praises for the past 10 years.

    • kykat

      today at 3:24 PM

      And my og comment getting downvoted on this very intellectual forum that definitely isn't an echo chamber

      • Petersipoi

        today at 3:47 PM

        Your very sarcastic, uninteresting comment getting downvoted is not an indication that forum isn't intellectual. It's an indication that you aren't behaving intellectually.

      • bflesch

        today at 3:58 PM

        Cognitive dissonance in tech millionaires is quite strong, still worth it to trigger them from time to time on a factual basis.

• aboardRat4

  today at 2:51 PM

  Big tech companies are always visited first by the G-men who need something done.