This really is a function of two things:

1) (Mainly) the huge increase in upstream capacity of residential broadband connections with FTTH. It's not uncommon for homes to have 2gbit/sec up now and certainly 1gbit/sec is fairly commonplace, which is an enormous amount of bandwidth compared to many interconnects. 10, 40 and 100gbit/sec are the most common and a handful of users can totally saturate these.

2) Many more powerful IoT devices that can handle this level of attack outbound. A $1 SoC can easily handle this these days.

3) Less importantly, CGNAT is a growing problem. If you have 10k (say) users on CGNAT that are compromised, it's likely that there's at least 1 on each CGNAT IP. This means you can't just null route compromised IPs as you are effectively null routing the entire ISP.

I think we probably need more government regulation of these IoT devices. For example, having a "hardware" limit of (say) 10mbit/sec or less for all networking unless otherwise required. 99% all of them don't need more than this.

• toast0

  today at 1:21 AM

  > 3) Less importantly, CGNAT is a growing problem. If you have 10k (say) users on CGNAT that are compromised, it's likely that there's at least 1 on each CGNAT IP. This means you can't just null route compromised IPs as you are effectively null routing the entire ISP.

  Null routing is usually applied to the targets of the attack, not the sources. If one of your IPs is getting attacked, you null route it, so upstream routers drop traffic instead of sending it to you.

  • martinald

    today at 1:45 AM

    Sorry, late here. You are right. I mean filter the IP in question.

  • today at 2:08 AM

• bsder

  today at 1:26 AM

  > If you have 10k (say) users on CGNAT that are compromised, it's likely that there's at least 1 on each CGNAT IP. This means you can't just null route compromised IPs as you are effectively null routing the entire ISP.

  How about we actually finally roll out IPv6 and bury CGNAT in the graveyard where it belongs?

  Suddenly, everybody (ISPs, carriers, end users) can blackhole a compromised IP and/or IP range without affecting non-compromised endpoints.

  And DDoS goes poof. And, as a bonus, we get the end to end nature of the internet back again.

  • lgeek

    today at 2:06 AM

    From having worked on DDoS mitigation, there's pretty much no difference between CGNAT and IPv6. Block or rate limit an IPv4 address and you might block some legitimate traffic if it's a NAT address. Block a single IPv6 address... And you might discover that the user controls an entire /64 or whatever prefix. So if you're in a situation where you can't filter out attack trafic by stateless signature (which is pretty bad already), you'll probably err on the side of blocking larger prefixes anyway, which potentially affect other users, the same as with CGNAT.

    Insofar as it makes a difference for DDoS mitigation, the scarcity of IPv4 is more of a feature than a bug.

    • zamadatix

      today at 2:59 AM

      (Having also worked on DDoS mitigation services) That "entire /64" is already hell of a lot more granular than a single CG-NAT range serving everyone on an ISP though. Most often in these types of attacks it's a single subnet of a single home connection. You'll need to block more total prefixes, sure, but only because you actually know you're only blocking actively attacking source subnets, not entire ISPs. You'll probably still want something signature based for the detection of what to blackhole though, but it does scale farther in a combo on the same amount of DDoS mitigation hardware.

    • spongebobstoes

      today at 2:19 AM

      you can heuristically block ipv6 prefixes on a big enough attack by blocking a prefix once a probabilistic % of nodes under it are themselves blocked, I think it should work fairly well, as long as attacking traffic has a signature.

      consider simple counters "ips with non-malicious traffic" and "ips with malicious traffic" to probabilistically identify the cost/benefit of blocking a prefix.

      you do need to be able to support huge block lists, but there isn't the same issue as cgnat where many non-malicious users are definitely getting blocked.

    • swinglock

      today at 6:52 AM

      You should block the whole /64, at least. It's often a single host. It's often but not always a single host, that's standardized.

      • vladvasiliu

        today at 7:09 AM

        Usually a /64 is a "local network", so in the case of consumer ISPs that's all the devices belonging to a given client, not a single device.

        Some ISPs provide multiple /64s, but in the default configuration the router only announces the first /64 to the local network.

        • TZubiri

          today at 7:22 AM

          Presumably a compromised device can request arbitrarily new ipv6 from the dhcp so the entire block would be compromised. It would be interesting to see if standard dhcp could limit auto leasing to guard reputation of the network

    • bsder

      today at 6:25 AM

      This DDoS is claimed to be the result of <300,000 compromised routers.

      That would be really easy to block if we were on IPv6. And it would be pretty easy to propagate upstream. And you could probabilistically unblock in an automated way and see if a node was still compromised. etc.

    • TZubiri

      today at 7:21 AM

      Better to rely on ip blocks than on NAT to bundle blocks.

  • ralnivar

    today at 2:14 AM

    I am a bit split this topic. There is some privacy concerns with using ipv6. https://www.rfc-editor.org/rfc/rfc7721.html#page-6

    Some time ago I decided for our site to not roll out ipv6 due to these concerns. (a couple of million visitors per month) We have meta ads reps constantly encourage us to enable it which also do not sit right with me.

    Although I belive fingerprinting is sofisticated enough to work without using ip's so the impact of using ipv6 might not be a meaningful difference.

    • miyuru

      today at 6:06 AM

      its hilarious that you have privacy concerns while at the same time using meta ads.

    • GoblinSlayer

      today at 7:17 AM

      Reportedly this is often incorrectly implemented, where /64 prefix is still a stable static address.

  • nine_k

    today at 2:28 AM

    Is there any money an ISP would make, or save, by sinking money and effort on switching to IPv6? If there's none, why would they act? If there is some, where?

    For instance, mobile phone operators, which had to turn ISPs a decade or two ago, had a natural incentive to switch to IPv6, especially as they grew. Would old ISPs make enough from selling some of their IPv4 pools?

    • ROBLOX_MOMENTS

      today at 2:40 AM

      They already lease them out. TELUS in Canada traditional old ISP rents large portion of their space to a mostly used for Chinese GFW VPN server provider in LA „Psychz“

      • TZubiri

        today at 7:25 AM

        The ISPs have to submit plans on how to use their IPs for the public,especially for IPv4, Arnic shouldn't approve this kind of stuff. Unless they lied in their ip block application, in which case they should be revoked their block.

    • rendaw

      today at 3:07 AM

      Presumably they'd lose money when a DDoS originating from their network causes all their ips to get blocked.

    • beeflet

      today at 2:39 AM

      less expensive IP space, more efficient hardware, and lower complexity if you can eliminate NAT.

  • rectang

    today at 2:05 AM

    Is it advantageous to be someone who supports IPv6 on a day like today?

• idiotsecant

  today at 2:41 AM

  Haha that last part is pretty wild. rather than worrying about systemic problems in the entire internet let's just make mandates crippling devices that China, where all these devices are made, will defffinitely 100% listen to. Sure, seems reasonable. Systems that rely on the goodwill of the entire world to function are generally pretty robust, after all.

  • saagarjha

    today at 2:52 AM

    If they don’t then the devices are not sold in the United States. It’s quite simple.

    • dylan604

      today at 4:39 AM

      Great to know that smuggling hardware into the US has been completely stopped.

      • morsch

        today at 6:07 AM

        If the analysis above is accurate, a few smuggled devices would not be an issue, as long as the zillions of devices sold at Walmart are compliant.

• gjsman-1000

  today at 1:31 AM

  > I think we probably need more government regulation of these IoT devices. For example, having a "hardware" limit of (say) 10mbit/sec or less for all networking unless otherwise required. 99% all of them don't need more than this.

  What about DDoSs that come from sideloaded, unofficial, buggy, or poorly written apps? That's what IoT manufacturers will point to, and where most attacks historically come from. They'll point to whether your Mac really needs more than 100mbps.

  The government is far more likely to figure it out along EU lines: Signed firmware, occasional reboots, no default passwords, mandatory security updates for a long-term period, all other applicable "common sense" security measures. Signed firmware and the sideloading ID requirements on Android also helps to prevent stalkerware, which is a growing threat far scarier than some occasional sideloaded virus or DDoS attack. Never assume sideloading is consensual.

  • ShowalkKama

    today at 1:59 AM

    >What about DDoSs that come from sideloaded, unofficial, buggy, or poorly written apps? That's what IoT manufacturers will point to, and where most attacks historically come from.

    any source for this claim? Outside of very specific scenarios which differ significantly for the current botnet market (like manjaro sending too many requests to the aur or an android application embedding an url to a wikipedia image) I cannot remember one occourence of such a bug being versatile enough to create a new whole cybercrime market segment.

    >They'll point to whether your Mac really needs more than 100mbps.

    it does, because sometimes my computer bursts up to 1gbps for a sustained amount of time, unlike the average iot device that has a predictable communication pattern.

    >Signed firmware and the sideloading ID requirements on Android also helps to prevent stalkerware, which is a growing threat far scarier than some occasional sideloaded virus or DDoS attack. Never assume sideloading is consensual.

    if someone can unlock your phone, go into the settings, enable installation of apps for an application (ex. a browser), download an apk and install it then they can do quite literally anything, from enabling adb to exfiltrating all your files.

    • gjsman-1000

      today at 2:03 AM

      Historically, it was called Windows XP and Vista about 15 years ago (Blaster, Sasser, MyDoom, Stuxnet, Conficker?). Microsoft clamped down, hard, across the board, but everyone outside of Big Tech is still catching up.

      Despite Microsoft's efforts, 911 S5 was roughly 19 million Windows PCs in 2024, in news that went mostly under the radar. It spread almost entirely through dangerous "free VPN" apps that people installed all over the place. (Why is sideloading under attack so much lately? 19 million people thought it would make them more secure, and instead it turned their home internet into criminal gateways with police visits. I strongly suspect this incident, and how it spread among well-meaning security-minded people, was the invisible turning point in Big Tech against software freedom lately.)

      https://www.fbi.gov/investigate/cyber/how-to-identify-and-re...

      > if someone can unlock your phone, go into the settings, enable installation of apps for an application (ex. a browser), download an apk and install it then they can do quite literally anything, from enabling adb to exfiltrating all your files.

      Which is more important, and a growing threat? Dump all her photos once; or install a disguised app that pretends to be a boring stock app nobody uses, that provides ongoing access for years, with everything in real-time up to the minute? Increasingly it's the latter. She'll never suspect the "Samsung Battery Optimizer" or even realize it came from an APK. No amount of sandboxing and permissions can detect an app with a deliberately false identity.

• devwastaken

  today at 2:47 AM

  1gb upload is extraordinarily rare.

  • saagarjha

    today at 2:51 AM

    It’s not; most places that give you gigabit fiber will give you a symmetric connection.

    • typpilol

      today at 3:57 AM

      Yup. Spectrum is Michigan will give you up to 2gbps down but not anything more than 200mbps up

      • dylan604

        today at 4:42 AM

        Is Spectrum fiber or DOCSIS? I didn't realize anyone was pushing these kinds of numbers for fiber. What's the point other than screwing the users?

        • today at 5:18 AM

    • vitaflo

      today at 3:07 AM

      Most places do not have fiber.

      • Dylan16807

        today at 3:15 AM

        We know. The problem is that the above comment said "extraordinarily rare" which is a very different and incorrect threshold.

      • dylan604

        today at 4:41 AM

        But for those that do...symmetric is the norm. The number of fiber connections is only going up.

      • ls612

        today at 3:42 AM

        This is probably technically true but very misleading. Fiber penetration in the US has been consistently rising for over a decade now and it is not at all uncommon to have either Google Fiber, Fios, or a local fiber provider available to you in a big city. I bet within the next decade most places will have gigabit fiber available.

        • _carbyau_

          today at 4:56 AM

          The US is a big place. But the world is bigger. The internet works across the whole world.

          There's a long way to go before fibre is commonplace across the world.

• nick32661123

  today at 12:56 AM

  Seems more likely that residential modems will be required to use ISP-provided equipment that has government mandated chips, firmware, etc to filter outbound traffic for DDoS prevention.

  • DaSHacka

    today at 1:02 AM

    Why should they be required to have hardware in their own network to filter that out when the ISP is obviously receiving all of their traffic anyway?

This is very challenging, in about one year the biggest recorded DDoS attack has increased from 5 Tbps to almost 30.

Almost all of the DDoS mitigation providers have been struggling for a few weeks because they just don't have enough edge capacity.

And normal hosting companies that are not focused on DDoS mitigation also seem to have had issues, but with less impact to other customers as they'll just blackhole addresses under larger attacks. For example, I've seen all connections to / from some of my services at Hetzner time out way more frequently than usual, and some at OVH too. Then one of my smaller hosting providers got hit with an attack of at least 1 Tbps which saturated a bunch of their transit links.

Cloudflare and maybe a couple of the other enterprise providers (Gcore?) operate at a large enough scale to handle these attacks, but all the smaller ones (who tend to have more affordable rates and more application-specific filters for sensitive applications that can't deal with much leakage) seem to be in quite a bad spot right now. Cloudflare Magic Transit pricing supposedly starts at around $4k / month, and it would really suck if that became the floor for being able to run a non-HTTP service online.

Something like Team Cymru's UTRS service (with Flowspec support) could potentially help to mitigate attacks at the source, but residential ISPs and maybe the T1s would need to join it, and I don't see that happening anytime soon.

• TZubiri

  today at 7:27 AM

  I'm surprised that the best response to ddos is not blocking traffic, but just handling it.

> “The problem is, even if those infected IoT devices are rebooted and cleaned up, they will still get re-compromised by something else generally within minutes of being plugged back in.”

In the year 2025, we should understand that such devices are defective. They should become bricks and companies that continue to sell such defective merchandise should fail.

• dylan604

  today at 4:44 AM

  Just wait 6 months, and the IoT vendor will go out of business and shut down the cloud servers which will effectively brick the device.

  • rectang

    today at 5:08 AM

    That may brick it for its intended use, but nothing's stopping a botnet from repurposing it.

• today at 2:20 AM

• TZubiri

  today at 7:28 AM

  How about some good old tort liability

• fukka42

  today at 2:34 AM

  Absolutely not. They should be patched and fixed.

  • rectang

    today at 2:44 AM

    So they get to ship a dangerous device that harms innocent third parties because they cut corners, but we’re supposed to reward them by doing the work to secure the devices they couldn’t be bothered with?

    • danparsonson

      today at 4:33 AM

      Patched and fixed by the manufacturer, was surely what GP was suggesting.

      Bricking the devices doesn't punish the supplier as much as it punishes the consumers.

      • rectang

        today at 5:04 AM

        Many of the companies that sold these devices have already bitten the dust, so they aren't even around to fix anything.

Are there any practical ways to monitor my home network traffic and detect if any devices on my network are compromised?

• tylerflick

  today at 1:52 AM

  Not sure about monitoring, but I always put any device I don’t trust on a jailed LAN/AP.

• bentcorner

  today at 2:22 AM

  A lot of home routers will give you a traffic graph - if yours doesn't you can either find one that does or flash/build one.

  I currently run opnsense which has an ok graph out of the box, I haven't fiddled with it to see if there's something fancy I could do here.

  I also used to use IPFire which was slightly clunkier but had a nicer usage graph.

• userbinator

  today at 1:44 AM

  Your ISP should give you a bandwidth usage meter.

  • skinner927

    today at 1:50 AM

    Haha. My ISP barely gives me an Internet connection

    • dylan604

      today at 4:45 AM

      You can rest assured your ISP can produce the graph. If your graph reaches a certain shape, they'll start shaping your traffic for you to something they prefer all while charging you the normal rate.

I'd rather there be periodic DDoS attacks, than a locked-down highly-regulated internet. Don't forget that infamous Franklin quote, and what Stallman has been warning us about for the past few decades.

I can already see the authoritarians salivating every time something like this happens.

• dylan604

  today at 4:47 AM

  > I can already see the authoritarians salivating every time something like this happens.

  Tinfoil hat theory says they do this intentionally so that the users demand stricter access willingly. Always better to have someone think it is their idea

Big botnet has nothing better to do than DDoS Minecraft servers?

• heinrich5991

  today at 8:35 AM

  Same happens to our online game: https://news.ycombinator.com/item?id=28675094. Not sure what it is about online games that attracts DDoSes.

• thenthenthen

  today at 2:28 AM

  …to sell ddos protection to the minecraft server admins, basically extortion.

  • dylan604

    today at 4:48 AM

    what about as an demonstration of their capabilities for someone else?

• shermantanktop

  today at 2:11 AM

  This is what I wonder. Must be fascinating to engineer such a massive distributed system, but at some point there’s no added value from another bazillion hosts in the network.

• crummy

  today at 3:20 AM

  I guess if they go after bigger targets they draw unwanted attention? Seemed odd to me too.

Seems pretty clear that the US needs strict regulation on any device connecting to the internet.

* no default password * * no login if not on the local wifi or wired ethernet *

• DaSHacka

  today at 1:03 AM

  Many manufacturers are already moving there of their own accord. I really don't think we'd need some legislation to fix this problem.

  • Eji1700

    today at 2:56 AM

    Ehhh I can see it. The right attack at the right time could directly or indirectly kill people, and that’s ignoring the fact it can cause economic havoc.

    Having the entire internet function on a “pay or be nuked” threshold that could easily get much worse if companies like cloudflare become less ethical (not that they’re saints).

• dehrmann

  today at 1:01 AM

  I'd rather the industry standardizes on some sort of guest network and proxy/hub. It could even ship with hardware from ISPs. Separating the network buys you a lot of security, and running everything through a proxy makes it easier to inspect data and creates a standard hook for using abandonware.

• today at 2:14 AM

9gigsofram was prolific in the "Minecraft Server era" (2010-2016).

Source: Server Owner's Chat

I'm honestly kinda curious why nobody's blocking these IPs from sending data near the source.

Like, I can come up with plenty of possible reasons, and reasons why it could potentially be very bad if ISPs started cracking down on this, but I don't actually know any reasons.

Are any talking about why / why not? It seems like this whole insecure-IoT-device thing would probably dry up pretty quickly if people's internet was cut off when one was detected. They can then turn around and lambast / sue / etc the company that sold it, putting pressure on the source of the problem. Right now there's no reason for sellers to do anything at all to ensure security, afaict.

So... not actually arguing in favor of it, but definitely curious about any stated ISP / core networking system's stated reasons.

• Mindless2112

  today at 12:01 AM

  > “The outbound and cross-bound DDoS attacks can be just as disruptive as the inbound stuff,” Dobbin said. “We’re now in a situation where ISPs are routinely seeing terabit-per-second plus outbound attacks from their networks that can cause operational problems.”

  ISPs are starting to feel the pain, so perhaps in the near future they will do something about it.

  • dloy

    today at 12:24 AM

    Perhaps, or perhaps not. Maybe if we held them accountable they would?

• MartijnBraam

  today at 12:27 AM

  This does happen, but it seems to depend on the ISP. In the Netherlands I've seen ISPs block the internet connectivity when they've detected infected devices, sometimes they send a letter before blocking and some ISPs seem to dump your internet connection in a captive portal. In all these cases it's been enough to call the ISP after finding the problem and you're connected again minutes later.

• kibbel

  today at 12:06 AM

  A large part of the article is dedicated to this, noting how disruptive it is to other services and customers, and listing a few countermeasures (detection and blocking at the ISP level, detection and blocking at the router level, and educating customers on not buying vulnerable IoT trash).

  • Groxx

    today at 12:12 AM

    Not really? At best it's "DDOS prevention sellers are having trouble" and "ISPs say they're doing fine". The vast majority of the article is talking about the various kinds of malware causing this, and how some have been "fixed" by stopping the individuals running it (which clearly doesn't work very well, new ones just fill the void).

    Or this:

    >“The crying need for effective and universal outbound DDoS attack suppression is something that is really being highlighted by these recent attacks,” Dobbins continued. “A lot of network operators are learning that lesson now, and there’s going to be a period ahead where there’s some scrambling and potential disruption going on.”

    Uh. No. That's gross negligence if they are only starting to think about it now - the trend has been clear for over a decade, and the IoT threat has been obvious since day 1 and even blasted over public news for the past few years. Their status is pretty much only one of: incompetent, malicious, or they have had plans but haven't acted on them fast enough or strongly enough for [some reason], and that reason isn't something I've seen. Surprises happen, prevention costs money and time, and there are plenty of reasons why everyone isn't already prepared for everything, so I think "incompetent or malicious" is pretty rare.... but what are those reasons?

• bombcar

  yesterday at 11:55 PM

  There's no economic incentive for YOU (as the proximate ISP) to do anything about it, it would cost money, and cost you customers.

  Any idea why they don't fix it?

  • Groxx

    today at 12:06 AM

    Yes, you generally see this kind of thing start from the pain-feelers and move up the chain to the pain-causers.

    So why hasn't that happened? These are clearly damaging to many, and ISPs are apparently doing next to nothing to prevent it, and it has been extremely clear for a while now that it's going to just become a bigger and bigger problem.

  • userbinator

    today at 1:46 AM

    Are there ISPs that don't charge customers for the amount of bandwidth they consume? Even "unlimited" has been ruled by courts to not really mean "unlimited", after all.

    • fukka42

      today at 2:42 AM

      Yes, most if not all of them. Is it different in the US?

      • bombcar

        today at 3:52 AM

        Most non-mobile ISPs will let you get pretty high on bandwidth usage before they flag you; and since DDoS are almost always relatively low-levels of bandwidth (on the source) it's unlikely you'd get flagged.

  • martinald

    today at 12:06 AM

    Of course there is. If you've got all your internet egress tied up with DDoS attacks from your network it is a big problem.

    • toast0

      today at 1:28 AM

      Most eyeball networks have a lot of inbound traffic and not very much outbound, but interconnections with other networks are almost always symmetric, so there's a lot of room for excess egress before it causes pain to the ISP.

      When I ran a large web site that attracted lots of DDoS, it didn't really seem worthwhile to track down the source and try to contact ISPs. I had done a lot of trying to track and stop people sending phishing mail under our name, and it's simply too much work to write a reasonable abuse report that is unlikely to be followed up on. With email, mostly people seem to accept the Received headers are probably true; with DDoS, you'd be sending them pcaps, and they'd be telling you it's probably spoofed, and unless I've got lots of peering, I'm not going to be able to get captures that are convincing... so just do my best to manage the inbound and call it a day.

    • bombcar

      today at 1:20 AM

      I think we’re just starting to see attacks that big - which might start some practical mitigations (or they’ll just upgrade transit).

• TZubiri

  today at 12:52 AM

  > They can then turn around and lambast / sue / etc the company that sold it, putting pressure on the source of the problem

  Or just unplug the culprit. But the key seems to be that the device continues working. Ideally you would just shutdown or disconnect the device. If fridge is infected, the fridge can still fridge, but it no longer has internet privileges.

  • quantummagic

    today at 1:00 AM

    Any device that participates in a DDOS needs to be recalled by the manufacturer, mandated by law. Make it potentially economically crippling to sell a vulnerable device, and security will be taken very seriously. Frivolous uses of tech, won't be worth the risk.

    • DaSHacka

      today at 1:06 AM

      This just in: every computer manufacturer forced to recall every single computer model they've ever sold because some users use weak passwords.

      I can't wait for all of them to switch to IOS-ified devices incapable of installing alternative operating systems or programs, as that would be the inevitable end solution for all these manufacturers if this was implemented.

      • quantummagic

        today at 1:20 AM

        Maybe that's a good thing; relying on users to choose good passwords is a cop-out. Systems should be safe-by-default. And owners losing their system if it participates in a DDOS, would add to the incentives to stop the nonsense. It persists because perpetrators, and those who unwittingly abet them, feel no consequences.

        • DaSHacka

          today at 1:35 AM

          At that point, you should force the pain on the individual themselves. Why should all of us be handicapped because there's a couple morons that can't set decent passwords and connect their devices directly to the internet?

          Even if the device removed the capability for passwords and used key based authentication, connecting it directly to the internet means if there's ever a vulnerability, all that was for naught anyway.

          • pabs3

            today at 3:37 AM

            This is the way, there should be no access by default, then on first access the user has to setup their desired authentication details, and if they want passwords, then they get a randomly generated one, not one they choose. There should also be a factory reset button too.

      • bee_rider

        today at 2:23 AM

        Does a weak user password have to provide remote access by default?

No evidence, just a wild guess, Aisuru might be tied to DDoS mitigation folks.

Isn't OVH known for offering relatively good anti-DDoS services? Why did they boot them?

"A spokesperson for Comcast responded, 'Currently our network is not experiencing impacts and we are able to handle the traffic.'"

In other words, attacks coming from our own IP's are not our problem ¯\_(ツ)_/¯

This doesn't seem very loving.

[dead]

Cool, can they try aiming it at twitter or truth please?