\

Tell HN: Camelgate NPM Outage (Cloudflare)

119 points - last Tuesday at 4:19 PM


EDIT: Back online?!

NPM discussion: https://github.com/npm/cli/issues/8203

NPM incident: https://status.npmjs.org/incidents/hdtkrsqp134s

Cloudflare messaging: https://www.cloudflarestatus.com/incidents/gshczn1wxh74

GitHub issue: https://github.com/sindresorhus/camelcase/issues/114

Anyone experiencing npm outage that's more than just the referenced camelcase package?

  • tom_usher

    last Tuesday at 4:38 PM

    Seems to be a change in Cloudflare's managed WAF ruleset - any site using that will have URLs containing 'camel' blocked due to the 'Apache Camel - Remote Code Execution - CVE:CVE-2025-29891' (a9ec9cf625ff42769298671d1bbcd247) rule.

    That rule can be overridden if you're having this issue on your own site.

      • internetter

        last Tuesday at 7:52 PM

        > any site using that will have URLs containing 'camel' blocked

        What engineer at cloudflare thought this was a good resolution?

          • Raed667

            last Tuesday at 8:03 PM

            I doubt the system is that simple. No one wrote a rule saying `if url.contains("camel") then block()` it's probably an unintended side-effect

              • keithwhor

                last Tuesday at 9:19 PM

                If this is a bet, I'll happily take the other side and give you 4:1 on it.

                  • dgfitz

                    last Tuesday at 9:28 PM

                    Me too.

                • ycombinatrix

                  last Tuesday at 10:13 PM

                  Akamai has been doing precisely that for years & years...

                    • last Tuesday at 10:23 PM

                      • benoau

                        last Wednesday at 12:36 AM

                        I think you can include advertising/privacy block lists in that vein too, although that allows for the users to locally-correct any issues.

                    • isbvhodnvemrwvn

                      last Wednesday at 4:34 PM

                      Judging by previous outages it was probably a poorly tested overcomplicated regex which matched to much.

                  • TacticalCoder

                    last Wednesday at 3:21 AM

                    [dead]

                • cbovis

                  last Tuesday at 5:05 PM

                  Confirmed here: https://www.cloudflarestatus.com/incidents/gshczn1wxh74

                  • oncallthrow

                    last Tuesday at 6:38 PM

                    WAFs are so shit

                      • ronsor

                        last Tuesday at 7:57 PM

                        WAFs are literally "a pile of regexes can secure my insecure software"

                          • mschuster91

                            last Tuesday at 9:07 PM

                            To be fair to WAFs, most are more than just a pile of regexes. Things like detecting bot traffic - be it spammers or AI scrapers - are valuable (ESPECIALLY the AI scraper detection, because unlike search engines these things have zero context recognition or respect for robots.txt and will just happily go on and ingest very heavy endpoints), and the large CDN/WAF providers can do it even better because they can spot shit like automated port scanners, Metasploit or similar skiddie tooling across all the services that use them.

                            Honestly what I'd _love_ to see is AWS, GCE, Azure, Fastly, Cloudflare and Akamai band together and share information about such bad actors, compile evidence lists and file abuse reports against their ISP - or in case the ISP is a "bulletproof hoster" or certain enemy states, initiate enforcement actors like governments to get these bad ISPs disconnected from the Internet.

                              • randunel

                                last Wednesday at 5:22 AM

                                Why would scrapes get blocked, is scrapping illegal?

                                  • eitland

                                    last Wednesday at 6:15 AM

                                    I don't know if it is, but I also don't think we are required to let dumb bots repeatedly assault or web sites if we can find a technical way to get around it.

                                    • Xylakant

                                      last Wednesday at 7:17 AM

                                      It's very often not, but it's still the website owners property and if they choose so, they can show misbehaving guests the door and kindly ask to remain on the other side (aka block them). Large scale scraping puts substantial burden on web properties. I was paged the other night because someone decided it would be a great idea to throw 200 000rq/s for a few minutes at some publicly available volunteer run service.

                              • cluckindan

                                last Wednesday at 11:26 AM

                                They do mitigate known vulnerabilities.

                                  • rcxdude

                                    yesterday at 9:02 AM

                                    They may mitigate known proofs of concept of vulnerabilities, and require a small amount of creativity to work around. At the cost of randomly breaking things.

                                      • cluckindan

                                        yesterday at 11:47 AM

                                        That creativity takes time. WAFs are the first line of defence, buying some time for fixing the actual vulnerabilities.

                            • UltraSane

                              last Tuesday at 11:40 PM

                              But are they less shit than the shitty software they filter traffic for?

                      • Recursing

                        last Tuesday at 4:28 PM

                        Any path with the word "camel" seem to trigger this: https://www.npmjs.com/search?q=camel | https://registry.npmjs.org/camel123 | https://registry.yarnpkg.com/camel456

                        Some discussion here https://github.com/npm/cli/issues/8203

                        Edit: this is resolved now https://status.npmjs.org/incidents/hdtkrsqp134s

                        • pvg

                          last Tuesday at 6:59 PM

                          This is not CF WAF's first rodeo https://news.ycombinator.com/item?id=20421538

                          Cementing its track record as a product that mostly doesn't do anything except for occasionally break the internet here and there to keep things fun and interesting.

                            • lynnesbian

                              last Wednesday at 12:37 AM

                              > a product that mostly doesn't do anything except for occasionally break the internet

                              I wouldn't say that. The postmortem you referred to links to another CloudFlare blog post - one about a pretty serious RCE vuln in Microsoft SharePoint that was blocked by their WAF: https://blog.cloudflare.com/stopping-cve-2019-0604/

                                • pvg

                                  last Wednesday at 3:29 AM

                                  I mean, it's hardly surprising CloudFlare will tell you this is a useful product. But it is to securing a web application what regex is to parsing HTML.

                                    • jiggawatts

                                      last Wednesday at 8:15 AM

                                      Sadly I work with web developers that all assume they don’t need to bother too much with security “because we have a WAF”.

                              • AdamJacobMuller

                                last Tuesday at 7:16 PM

                                I'm not sure why "WAF has false positives" makes it useless, nor would I say this is anywhere near the scale of "breaking the internet" and I'm not even fan of the concept of WAFs in general.

                                  • pvg

                                    last Tuesday at 7:31 PM

                                    The last one took out a lot more stuff than this one but the argument is the same - this product is a checkmark thing and when it's not fulfilling its checkmark purpose, it causes outages. Still an amusing bi-modality! I suppose it shares it with DNSSEC.

                                      • misiek08

                                        last Tuesday at 7:48 PM

                                        Basically CF default WAF settings saved more small and medium companies I can even count to. I’m not CF fan, but WAFs (with rate limiting) do help. Sad that one or two incidents for that complicated and big services make people post such comments, but cmon - it doesn’t have AI in it's name so sheeps have to cry, right?

                                • calvinmorrison

                                  last Tuesday at 7:04 PM

                                  we've used it to rescue some vintage appliances that are basically unsecurable.

                              • nwalters512

                                last Tuesday at 4:50 PM

                                The npm folks have officially acknowledged an incident now: https://status.npmjs.org/incidents/hdtkrsqp134s

                                • miyuru

                                  last Wednesday at 3:36 AM

                                  Outsourcing WAF is a double-edged sword.

                                  I would have thought a large company like GitHub or Microsoft can have their own WAF team for their apps.

                                  (NPM is owned by GitHub, and GitHub is owned by Microsoft)

                                  • klysm

                                    last Tuesday at 6:54 PM

                                    This is what you get when you buy security as an add-on product

                                      • troyvit

                                        last Wednesday at 3:14 PM

                                        Some orgs can't afford not to.

                                    • mplanchard

                                      last Tuesday at 6:30 PM

                                      Glad you posted something, thought I was going nuts

                                      • time4tea

                                        yesterday at 6:12 AM

                                        Scunthorpe problem

                                        • drusepth

                                          last Tuesday at 6:58 PM

                                          Is this also why unpkg has been up and down all morning?

                                            • ycombinatrix

                                              last Tuesday at 10:15 PM

                                              unpkg barely works even when there's no incident

                                          • last Tuesday at 6:10 PM

                                            • last Tuesday at 4:58 PM

                                              • last Tuesday at 4:38 PM