I'm surprised neither OpenAI nor Anthropic made this move first. The Chinese open weight models are pulling ahead and commoditizing their value proposition.

Baking models onto silicon would've been the next logical move to get a moat.

Google is already doing this and has an experimental project on top of already having TPUs and cramming their quantized flash onto individual TPUs for inference.

• anthonypasq

  yesterday at 9:56 PM

  Personally I think Apple should have acquired them. if you could burn a gemma4 class model into an iphone and actually get extremely low latency and low battery usage it would feel like the future IMO. even if it means you wont get frontier intelligence, there might actually be incentive to buy a new mobile device every year again.

  • Melatonic

    yesterday at 10:34 PM

    The Taalas chips are not physically small. And part of their secret (if you look at the design) is just locating a bunch of memory soldered on the edges ( I belive higher amounts of SRAM ? )

    • chorizo

      today at 12:39 AM

      Baking the base models on to ROM makes a lot of economic sense. SRAM for the KV cache & fine-tunes, not so much. Sure you’d get incredible speeds but it’s not scalable from a die-size or cost perspective.

      Rather base model on ROM + KV cache on DRAM is much more scalable. Also this would work great for edge devices that have a 2-5 year lifecycle.

      • adrianN

        today at 3:00 AM

        It is my understanding that just baking the model itself into silicon only gives moderate gains because memory bandwidth remains a bottleneck.

        • chorizo

          today at 3:07 AM

          The big benefit is ROM cells require fewer components than DRAM. So the chips would be tiny, dense, cheap and consume far less power.

          • klodolph

            today at 3:38 AM

            I thought DRAM was pretty dense already. Is mask ROM that much denser?

            • chorizo

              today at 3:50 AM

              Yes, each rom bit can be a transistor or even a diode with a decoder circuit. Simplest Dram cell is capacitor+transistor - and you need a clock, refresh circuit etc.

              Someday, I imagine model weights could even be encoded as analog resistors (memristors or similar) for even greater density

              • ReactiveJelly

                today at 4:41 AM

                Hm. I wonder how many relays I'd need to make a physical MNIST classifier. That'd be dope

    • selcuka

      today at 1:08 AM

      Their PoC chips are big, but then it's ridiculously fast (have you seen chatjimmy.ai?). Also they must be holding a bunch of patents.

  • adgjlsfhk1

    yesterday at 10:12 PM

    I don't think this works out from a cost/silicon perspective. Small models already run pretty well in software (since the weights fit in cache) and big models require silicon area proportional to the size of weights. On a mobile device putting a chip like this is competing directly in BOM and power against a whole lot more l3 cache, and the l3 cache makes everything faster

    • trebligdivad

      today at 12:46 AM

      What, even if it means you can run models without relying on the currently backlogged DRAM production?

      • adgjlsfhk1

        today at 2:18 AM

        The size of model we're talking about running doesn't need much if any dram.

    • teaearlgraycold

      yesterday at 10:36 PM

      My question is what changes about LLM use cases when you’re getting 1000 tok/s? Models in silicon might dramatically change how we think about them.

      • QuiDortDine

        today at 12:24 AM

        Did you use chatjimmy? It's somewhat terrifying to use when you think of the potential results with a better model.

        Ok, real life example: I now spend most of my time, as a developer, waiting for the agent to do its thing (after careful prompting, I'm also thinking about work stuff, don't worry I'm not useless). What if it gave back the same excellent results, but instantaneously? Why, then, I certainly would become the bottleneck. So, quite possibly, my last work task would be to plug this agent directly into the ticket system where the domain experts input their feature requests. Maybe we still need 1 developer out of 100, to coordinate releases and all that (ok, say 1 out of 10).

        But that's not taking things far enough: why do we need these domain experts at all? Our pitch is clear, and all software-enabled, though it took years to develop. We can just have the clients express their concerns to the AI, directly or indirectly. Have multiple lighting-fast agents with different roles (refactoring agent, new features agent, debugger agent, domain expert agent, etc.). So we fire everyone, maybe keep 1 product owner / devops to keep the trolls out. The cost is still probably 100 times less than it used to be (beyond the initial cost of acquisition of the magic machine or whatever).

        But one of these clients, surely, will realize that these 10 years of manual and slowly-automated development can now be emulated in very, very little time. Why not just, say, take screenshots of the entire app and feed them into the magic machine? Why, this way, they could have the service for a tenth of the yearly cost, forever!

        And then the economy implodes.

        I'm not saying it's THE most likely version of things, I'm saying that at a certain level, quantity (or rather, speed) is a quality all its own. And this new quality might change the world. Let's hope it's for the better!

        • pastel8739

          today at 4:02 AM

          This is the same pitch that people make about AI today. Speed isn’t the differentiator, quality is

        • xur17

          today at 1:14 AM

          I'm not sure inference speed is always the slowest thing for me right now. The agent is running tests, loading webpages, etc, which all take time. I don't know if a fast agent would speed things up in all cases.

          That said, it obviously depends on the project.

          • jodrellblank

            today at 1:32 AM

            > "The agent is running tests, loading webpages, etc, which all take time"

            A frustrating vision of the future would be when we've been asking for faster loading lighter web pages for years and then companies start caring about it and improving it not for us humans but for LLMs.

            • evilduck

              today at 2:32 AM

              It's already kind of that way with MCP servers popping up everywhere. The JIRA MCP server is like a couple orders of magnitude faster to work with than the website itself.

            • andersonpico

              today at 2:56 AM

              They finally cared about clear requirements and documentation when that meant getting rid of devs.

            • layla5alive

              today at 3:35 AM

              That happened at corpo work for each of: * Build times * CI latency * Developer tooling * Documentation * Modularity

        • bdangubic

          today at 2:27 AM

          > I now spend most of my time, as a developer, waiting for the agent to do its thing (after careful prompting, I'm also thinking about work stuff, don't worry I'm not useless).

          you need to launch 10-15 more terminals, who is waiting these days? :)

        • wsintra2022

          today at 12:49 AM

          I think this reads like Ray Kurzwheil (sorry not able to spell that off top of my head, that bloke who wrote that book about the future) .. But yeah very dystopian and totally realistic. Not if but when..

          • QuiDortDine

            today at 12:53 AM

            I LOVE Kurzwheil! Thank you for the compliment, I'm very far from having his writing skills. But yes sci-fi is looking more and more like, well, sci.

      • HDBaseT

        today at 12:04 AM

        In the case on on-device/self-hosted LLMs. You ask your agent to implement xyz feature 10 times and use a model to compare the outputs and combine the best results.

        Raw intelligence becomes slightly less important when you can iterate and improve automatically. You can still claim it was "one shot" even when 30 different implementations were made then combined.

      • in_a_society

        yesterday at 11:22 PM

        The best way I can explain it is that it's the same feeling when I upgraded from 56k dialup to cable broadband.

      • LarsDu88

        today at 4:43 AM

        Massive economic simulations with thousands if not millions of agents to front run the global economy and stock market.

        Fully interactive realtime NPCs in videogames at scale.

        Recommender systems that simulate individual consumers.

        Crazy shit

      • RussianCow

        yesterday at 10:43 PM

        That likely isn't as relevant for on-device iPhone usage as it is for Real Work™. I won't notice the difference between 50tps and 1000tps when asking Siri a question.

        • spijdar

          yesterday at 11:11 PM

          I don't know. As others have said, the Taalas chip wasn't small, or particularly low power, so it's hard to "imagine" what that tech in an cell phone chip might look like.

          But if the basic premise of "good enough LLM at insane throughput" holds, I think it could qualitatively change local uses of LLMs. At a certain speed point, you're able to move from request -> response to a cascade of tool calling and "subagents", which could allow a small model to be much more useful, if provided with a lot of local data and tool calls.

          That said, this is assuming you could stuff a "good enough" model into a phone with Taalas-like technology. The Taalas tech demo was an 8B parameter model and required hundreds of watts (IIRC) to run. The efficiency was good given the speed (as I understand), but it's not clear at all that the approach scales small enough to be a sensible coprocessor on an iPhone or whatever.

          • intrasight

            today at 12:23 AM

            Box that plugs into my desktop would be fine. Or perhaps in SSF form factor.

        • retatop

          yesterday at 11:20 PM

          But wouldn't higher tps allow for more reasoning or other hidden processes, potententially making a smarter model?

          • dabbz

            yesterday at 11:58 PM

            This is my thought as well. Models have to be intentional about which tokens they burn because there's a real lag time. If you can just fork out 10 different reasoning sessions at once with no regard for token waste/lag, you can compensate a smaller model with just doing more at once with it. No idea if this is reasonably true though.

        • nvme0n1p1

          yesterday at 11:22 PM

          That order of magnitude could be the difference between "the users wants me to open the notes app, let's open it" and "I've scanned all your notes before you could blink and found what you're looking for".

        • p1esk

          yesterday at 11:39 PM

          If Siri is using a 3T model in high reasoning mode to answer your question you will.

    • dboreham

      today at 12:12 AM

      Works great from a press release perspective though.

    • bastawhiz

      yesterday at 10:39 PM

      The weights might fit in cache, if you're using a small model. If you wanted to have a 20B+ parameter model, that's just going in RAM. You could put more RAM in the device and pay the perf cost or have a dedicated chip. Most devices already have a dedicated chip, this just changes which silicon you're spending the money on.

      • wmf

        today at 12:07 AM

        That math doesn't really work.

        8B model (FP4) = 4 GB DRAM = 32 Gb DRAM = 80 mm2

        8B model (Taalas) = 4 GB ROM = ~800 mm2

  • bsaul

    yesterday at 9:59 PM

    That's actually a really good point... There's currently zero incentive to buying more hardware, and that's one very good reason do have a new one.

    • sebular

      yesterday at 11:13 PM

      But this is already happening with iPhones. Apple is touting on-device AI and only the latest phones offer the full capabilities. Newer phones will be able to run better models, so the incentive is there as soon as someone makes the killer app that only makes sense when the model is running locally on your phone.

  • superb_dev

    yesterday at 10:07 PM

    From what I remember, these chips are not mobile size yet

    • bradfa

      yesterday at 10:11 PM

      A small model would be. I think that’s more the point. It’s definitely not SOTA but it’s fast and energy efficient and local.

      • mdp2021

        yesterday at 10:27 PM

        > A small model would be [mobile size]

        A ~30mm side for the HC1 tech for an 8b model (still unclear the planned HC2)?

        • teaearlgraycold

          yesterday at 10:37 PM

          Is that analogue or are they baking floating points into the silicon?

          • AlotOfReading

            today at 12:20 AM

            It's entirely possible they're using something like block floating point, where most of the hardware is simply fixed point. AMD's NPU does this, for example.

      • wmf

        yesterday at 10:31 PM

        Nope, a small model would be larger than the whole iPhone SoC.

  • makeitdouble

    yesterday at 11:29 PM

    Slightly besides your point, but it's interesting how many here naturally ponder about how the current winner could or "should" keep winning, instead of how another company could become a competitor by doing the more clever thing the incumbent isn't thinking about.

    • krisoft

      today at 12:20 AM

      It is not a “should”. At least not in the “we wish it were so” sense.

      It is more that there are multiple reasons why this idea (burning an LLM into silicone and deploying it into a device in people’s pockets) requires huge piles of cash and the kind of engineering chops only a few company posesses.

      Of course i would like it if a small upstart would do this, but it doesn’t seem likely as a posibility. They won’t have the funds to fab the IC. They won’t have the funds to train and validate the model before burning it into silicone. They can’t absorb the risk of the first tape out going wrong. They can’t absorb the risk of the model being faulty in some subtle way. They don’t have a device to integrate the IC into. They won’t have the funds to develop one. If they somehow would make a device they don’t have the marketing and sales channels built out to get the device into people’s hands in sufficient numbers to justify the development cost.

      Basically this idea feels ruinously expensive. Apple has deep pockets, they already have working well-regarded phones, and an ethos of privacy preserving innovation. This is why this idea feels well suited for them and not many others.

      Do i want the winners to keep winning? No. But not many others can pay for a moonshot crossed with a manhattan project. They just can’t.

  • whatsThisBtn4

    yesterday at 10:34 PM

    Apple is somewhere between fashion company and second rate tech company.

    They could have 9 year old AI and still post profits.

    Not sure if it's my pixel or android, but I made a randos jaw drop with what the crappy AI on android can do.

    When are we getting android OpenClaw?

    • today at 3:05 AM

• unsigner

  today at 4:56 AM

  Their thing is improving the models; it would be extremely counter-company-culture to bet on models plateau-ing. Maybe wise in terms of hedging, but still difficult to pull of as a company decision.

• moshun

  yesterday at 9:45 PM

  Considering the rate of model development and rail hopping, seems like baking models into silicon is speed-running obsolescence.

  • LarsDu88

    today at 4:46 AM

    It depends on how quickly you can bake new architectures.

    Text diffusion might be a disruptor here, but let me just say the most cutting edhe form of image diffusion (JiT and DiT) right now is just a big fat stack of alternating attention and MLP matmulls. Not theoretically hard to bake

  • breuleux

    yesterday at 10:19 PM

    If you’re only running models for frontier capabilities, yeah. For tasks where current models are smart enough, running them 100x faster is the most impactful improvement you can make. Consider all the things you could use a model for, but don’t, because the latency is just a bit too high.

  • zxspectrum1982

    yesterday at 10:45 PM

    I'd gladly pay for a Claude Opus 4.6 Thinking High in silicon and use it for 1-2 years. It's good enough for many coding tasks.

    • subroutine

      yesterday at 11:48 PM

      But Claude Opus 4.6 is not really practical. Taalas' process seems targeted for edge models. Their proof of concept model, for example, is a heavily quantized version of Llama 3.1 8B and even then they acknowledge their custom 3-bit/6-bit representation causes model quality degradation.

      Taalas is going to have a tough time putting a trillion-parameter model on one conventional die. Their HC1 die is already near the maximum size that conventional lithography can expose. They claim they could partition the model across many chips, but I'm not sure if they have tested this process or what it means for compute. The basic storage arithmetic is unforgiving: for a one trillion parameters model at four bits it will take 50–100 chips. To service a sizable customer base will take thousands of 100-chip fabs.

      That all said, I'm bullish on this technology, and look forward to seeing it evolve.

      • vatsachak

        today at 12:31 AM

        Yeah. But this kinda feels like a bandaid.

        Eventually someone will have to solve compute in memory at scale.

    • andix

      today at 12:20 AM

      With thousands of token per second output it would be an enormous waste of resources. Such chips are clearly made to process thousands of conversations simultaneously. Not necessarily in parallel. All LLM workflows are turn based right now, there are often seconds between turns until tool calls finish or users type the next message.

      If the LLM response only takes a few milliseconds, the chip can process hundreds of other requests until the first conversation becomes active again.

    • NiloCK

      today at 2:55 AM

      Not so long ago, I was good enough for many coding tasks. But I found that things can change in a hurry.

      Yes, a cheap and fast Opus4.6 can drive a lot of value in current context. But if we continue to craft bigger-and-bigger balls of mud, Opus 4.6 may end up hitting its conceptual ceiling and unable to contribute.

      Winding the clock back on your statement gives:

      > I'd gladly pay for a Claude Sonnet 3.5 in silicon and use it for 1-2 years.

      Man, I dunno.

      • kennywinker

        today at 4:04 AM

        Assuming moore's law like progress, which I'm 100% sure isn't going to happen - I think we're at the top of the S curve already. But assuming dramatically increased intelligence every year this is still the exact same position as anyone who bought a computer in the last 5 decades. Yet, people did very much buy computers.

    • Gigachad

      yesterday at 11:09 PM

      It costs something like $300,000 for the hardware to run a model of that size. You'd pay that for a single model for 1-2 years? Not even the AI companies can justify that kind of spend which is why they keep extending the expected lifespan on their hardware in the accounting.

      • zxspectrum1982

        yesterday at 11:11 PM

        I'm expecting the Taalas MSIC version to cost a fraction of that. Then probably have some kind of cheap subscription to Anthropic for updates (yes, Taalas chips can receive a certain kind of updates: they have a small SRAM).

      • mdp2021

        today at 12:03 AM

        > It costs something like $300,000 for the hardware to run a model of that size

        You did not compute that as the cost for a speculative card from Taalas, right?

        • Gigachad

          today at 1:10 AM

          It's the cost of the current nvidia hardware used to run these models. Of course all bets are off if you are accounting for some future chip that doesn't exist yet which could cost less.

  • nowittyusername

    today at 2:13 AM

    Depends on how much it costs the consumer. If I could buy a "cartridge" of Kimi K3 for 300 bucks I 100% would buy that shit asap. Even if it's "no good" after lets say 4 months still would be worth it IMO.

    • desmaraisp

      today at 2:23 AM

      That's definitely super-enthousiast territory. Paying 80 bucks a month for AI is more than 99.99% of people would be willing to do

      • nowittyusername

        today at 2:43 AM

        This will be considered very cheap within the year IMO. The value you get from AI is exponentially increasing and like all tech just takes some time to ramp up. Cell phones, internet and many other amenities when they came out many people were not willing to pay for but that all changed and considering how important AI tech is this will also be the case especially considering if its 100% private such as for that cartridge.

      • kennywinker

        today at 3:58 AM

        That's because the super-enthusiast will upgrade in 4 months when a better model is released. The casual user would keep it for years. A year of claude at the lowest plan is almost $300

  • topspin

    yesterday at 10:18 PM

    "seems like baking models into silicon is speed-running obsolescence"

    Now maybe. When models are flying passenger aircraft, other prerogatives will assert themselves. When a 50TB ROM means you can impulse purchase a ChatGPT 6.3 xhigh that runs on batteries, yet more use cases will be apparent.

    • mdp2021

      yesterday at 10:34 PM

      Well, 50TB ROM Taalas HC1 style would be apparently a 400000b transistor system through a chip sized 2.5 meters on the side... :)

      • preg_match

        today at 1:43 AM

        Yes but have we considered employing, like, a really big block of ice? Like old-timey surgeries? What if we put a big block of ice on the 2.5 cubic meter CPU what happens then?

      • thfuran

        yesterday at 11:06 PM

        Phones were getting too thin anyways.

    • heywoods

      today at 12:08 AM

      Or autonomous weapon systems, missiles, and drones.

      • umeshunni

        today at 1:22 AM

        Why would they need multi TB frontier models?

  • ray_v

    yesterday at 9:55 PM

    I could see this making sense when model development start to settle down ... it's going to settle down, right? ...

  • amelius

    yesterday at 9:52 PM

    Not sure. You can fix the transistors but leave the connections between them open for flexibility, so you only need to change the manufacturing process for the upper masks for every new model.

    • tsujamin

      yesterday at 10:16 PM

      Surely that added flexibility negatively impacts the density/parameter count of the model you could etch?

    • sroussey

      yesterday at 9:58 PM

      Or do a hybrid

  • mdp2021

    yesterday at 10:05 PM

    Compute the cost of producing n of them devices, imagine a fair price based on that, and see if that local, blazing fast card* can be an asset that could be replaced periodically.

    *(It's local: private files managing firm oriented. It's blazing fast: it can be placed into recursive, intensive local workflows.)

  • alightsoul

    yesterday at 10:10 PM

    Which is exactly what companies and shareholders want to increase sales.

  • flyinglizard

    yesterday at 10:07 PM

    Look at it the other way: compared to the cost of training a model, the cost of making a custom ASIC is trivial.

  • try-working

    yesterday at 10:45 PM

    obsolescence is the whole point. apple gets to sell a new phone very 6-12 months because of it.

    i have written about this:

    "For device makers

    Packaging models with laptops and smartphones will let application access near free, low latency inference and potentially offer users a better experience with the option of preserving data on-device. This is viable under the condition that tasks that do require larger expert models that run in the cloud can be routed to external models. A side-effect of local models and what will let Apple cut upgrade cycles from ~4 years (?) down to 12-18 months is specialized hardware to run them. For almost a decade, smartphones have been trying to compete on better cameras. This coming decade will see them selling better GPUs, NPUs, ASICs and whatever other things they'll be calling the inference chips, to drive re-purchase. Every six months will see a better model on new hardware, which will enable better performance in certain applications."

    https://try.works/role-model-the-case-for-a-model-routing-pr...

    • nomel

      yesterday at 11:05 PM

      No, the point is inference speed and power.

      • try-working

        today at 4:01 AM

        you don't understand what I wrote.

• wraptile

  today at 3:29 AM

  This seems like a very bad and dangerous direction for our society.

• throwaway27448

  yesterday at 11:23 PM

  You need to find customers for several-generations-ago models before this makes any sense. AMD is a lot more incentivized to look than mr vanilla llm is

• giancarlostoro

  yesterday at 10:28 PM

  ASICs is what took over Bitcoin mining, cheaper in all ways, and lasts longer than Nvidia GPUs for inference.

  • SR2Z

    yesterday at 11:14 PM

    > cheaper in all ways,

    Bitcoin mining doesn't have large memory requirements, but does have huge compute requirements. ASICs work great there because it's very straightforward to add some circuits for computing hashes. If you _also_ have to add many GB of memory, then suddenly ASICs will cost as much or more than comparable off-the-shelf hardware and they won't be faster unless you've also invested in huge memory bandwidth.

    • giancarlostoro

      yesterday at 11:40 PM

      My understanding is an ASIC can last 10+ years, where are Nvidia enterprise GPUs are rated for 5...

      • SR2Z

        yesterday at 11:56 PM

        Most enterprise GPUs are scrap after 5 years because they're so inefficient compared to newer models. It's entirely possible to make them last longer by undervolting them, people just don't because it doesn't make sense.

        Bitcoin OTOH has used the same PoW algorithm for a decade. Barring some really exciting discoveries about the nature of computation, new ASICs are not that much more efficient than old ones.

        BTC mining is also not exactly competitive anymore; the nature of the PoW algorithm means that it's dominated by a few large players who've set up shop next to a dam and who pay very little for electricity.

        New entrants are highly discouraged because the mining rewards are constantly halving, it's hard to find cheap power, and the price of BTC is now so volatile that a yearslong investment is very likely to lose money.

• mrtksn

  yesterday at 10:02 PM

  Isn’t that kind of useless for the stock? It sounds complicated, unlike having number of CPUs go up.

  It’s like talking about anything else than Megapixels when everyone was convinced that megapixels must go up in certain periods of the smartphone boom.

• LPisGood

  yesterday at 9:45 PM

  I’m surprised Nvidia hasn’t partnered to make a Claude chip yet. It’s a win/win you can license them out, sell them when they become obsolete, etc.

• CircuitSeuss

  yesterday at 10:30 PM

  Apparently Anthropic is moving that way: https://arstechnica.com/ai/2026/08/anthropic-confirms-plans-...

  • mdp2021

    yesterday at 10:57 PM

    Not necessarily: it is relevant to Taalas only if it is a compute-in-memory architecture.

    The Jalapeño mentioned («Anthropic is not alone in walking this path») in the article is still a classical Von Neumann architecture.

    And Taalas' idea makes sense in a perspective of scale - producing a large number of cards; "for internal use" (a lower order of items) means a high production cost.

• UncleOxidant

  today at 12:46 AM

  I guess I'm not understanding why this makes sense for AMD to buy Taalas unless they plan to get into hosting. It doesn't seem like a great fit.

• la6479

  yesterday at 11:39 PM

  Just to see how fast it is try chatjimmy.ai

  • tasty_freeze

    today at 1:51 AM

    It is really fast and ... really hallucinates. I asked "Does the Wang corporation still exist? If not, what happened to it?" and it replied (in part):

    "Yes, the Wang Corporation, the company that originally developed and marketed the Wang 2200 computer, still exists as a rebranded company under the name PPL (Precision Pencil and Label), but it has undergone significant changes and challenges over the years.

    Here's a brief overview of what happened:

        Founding and Growth: The Wang Corporation was founded by An Wang in 1969."
    

    In fact, Wang labs was founded in 1951. PPL seems to be a made up entity. But it did generate those "facts" in 0.033 seconds. If people value speed over accuracy then I can write an LLM that is 100x faster than chatjimmy.ai and make big bucks by responding one of N canned responses to any question.

    • mickaelkerjean

      today at 2:34 AM

      their tech is a mere demo to open up a new path, the day we can have some asics running a Qwen3.6 27b, this would open up new doors

  • mr_mph

    yesterday at 11:40 PM

    Pretty incredible to see. It reminds me of when I first used the Groq chatbot, except in this case it's a full response instantly.

• stingraycharles

  today at 2:00 AM

  Didn’t Anthropic acquire Cerebras? Seems like a move into the same direction.

  I also think that etching models into ASICs may be a bit too inflexible for what OpenAI and Anthropic want.

  • wyrdcurt

    today at 4:32 AM

    No, that's backwards. OpenAI are the ones investing in Cerebras. Part of the deal is that they can't sell to Anthropic.

• alightsoul

  yesterday at 10:09 PM

  Because Openai and anthropic are not hardware companies. They outsource that to Broadcom and AWS' Annapurna labs.

  • wmf

    yesterday at 10:33 PM

    OpenAI and Anthropic are both designing ASICs.

    • alightsoul

      yesterday at 10:49 PM

      So they have decided that putting a small LLM on a phone would backfire because people would have a negative perception of their cloud models. Pretty sure AMD will use these taalas chips in data centers, not phones

• karmasimida

  yesterday at 10:01 PM

  A model can't be updated, and a chip that is only relevant for 6 months at max?

  • anigbrowl

    yesterday at 10:28 PM

    Depends what you mean by relevant. If you use AI primarily as a search/knowledge engine, it makes no sense. If it's your capable assistant that has a lot of general knowledge, can do tool calls, and has a big context window, very doable.

    Indeed, for some kinds of applications involving secure/legal data etc. I can see the consistency of silicon winning out, because it combines performance with immutability and guardrails in hardware. Some chips have write-once PROMs to store password hashes and similar, you could do the same thing with prompt hashing to absolutely force or forbid certain behaviors. A model that can't be updated is also a model that can't be hacked.

  • askvictor

    yesterday at 10:09 PM

    People already buy new phones every year, this just creates even more reason to do so

    • Gigachad

      yesterday at 11:10 PM

      Outside of this website I've never met a person who buys a new phone every year. It's closer to every 3-4 years for most people.

      • boelboel

        yesterday at 11:23 PM

        Closer to every 5-6 years these days and with ram prices going up it will be even longer. Especially with the low/mid range phones, which are most phones outside some developed countries, people will keep their phones as long as they can.

        • Gigachad

          yesterday at 11:36 PM

          Would depend on the income levels, but yeah, buying a new phone these days is entirely a non essential luxury. An iphone easily lasts 7 years so the moment money is tight, it's a very easy choice to not buy a new one.

    • throwaway240403

      yesterday at 11:25 PM

      Your location/income bias is showing. Most people do not buy new phones every year.

      • winrid

        today at 2:17 AM

        I live in the bay area and buy a phone maybe every 3 years? Why do people waste so much money :D

  • simpsond

    today at 1:19 AM

    Base model sure, but the stack will be hybrid. It’s still early days here. Too bad FPGAs have such large feature size.

  • hamdingers

    yesterday at 10:47 PM

    One of these chips smart enough to take orders at a drive-thru would be relevant for a decade, minimum.

• wolttam

  yesterday at 9:57 PM

  It's a terrible moat. You etch the silicon then nobody wants to run it in 6 months because models have advanced that much further.

  • nine_k

    yesterday at 10:05 PM

    Not so if it's embedded in something smart enough for its intended purpose.

    Think vision, spatial reasoning, speech synthesis, even some speech analysis. Think self-driving cars (and drones) that need 10x less power for the brain, and can think at 10x situation per second.

  • anigbrowl

    yesterday at 10:30 PM

    This is only true for people who are solely focused on performance. There is absolutely a market for acceptable performance combined with predictability.

    • teraflop

      yesterday at 11:42 PM

      True, but predictability cuts both ways.

      We're all used to having to constantly update our browsers and phones to keep up with the security arms race. If a frozen model can't be updated, it will predictably remain vulnerable to any "exploits" or idiosyncratic quirks that people discover over time.

      Let's say, as somebody suggested in another comment, that you buy 100,000 of these chips and deploy them to run fast-food drive-thrus. And then somebody discovers the model has a fondness for goblins[1], and if you role-play convincingly enough, you can get it to accept payment in shiny buttons and rodent skulls instead of cash.

      What do you do then? I guess your options are to try and fix the behavior with a better prompt, or put some kind of filter in front of the model to catch attempted exploits. If the filter is cheap and dumb it probably won't work well enough, and if you use another model as a filter, you've negated the cost and speed benefits of putting the first model in hardware.

      Of course the real answer is to just never expose the model to situations where an adversarial input could possibly lead to an undesired output. But that drastically limits what you can do with it.

      [1]: https://openai.com/index/where-the-goblins-came-from/

      • anigbrowl

        today at 2:08 AM

        I see your argument but your example seems highly contrived. I can't think why you'd want to use something like this for something as dynamic as takeout ordering, where you might have to deal with bad customers, supply chain breakages, public health recalls, or any of many other probabilistic events.

        I think it's far more likely to see them used in safety critical applications where you need a capable model that can run on low power and doesn't have multiple layers of operating abstractions between the model and the hardware.

        • fwip

          today at 3:01 AM

          What safety critical applications would be a good fit for LLMs?

      • noisy_boy

        today at 1:53 AM

        > Of course the real answer is to just never expose the model to situations where an adversarial input could possibly lead to an undesired output. But that drastically limits what you can do with it.

        Does it though? Isn't that what CPUs are, very fast-not-so-clever computing brain surrounded by layers that protect it?

  • yesterday at 10:11 PM

  • speed_spread

    yesterday at 10:03 PM

    If a model is good enough today, it's still gonna be good enough in a year. Except you'll be able to serve it 1/100 of the price. Or 100x the speed.

  • twobitshifter

    yesterday at 11:01 PM

    OTOH, people get a new iPhone every year and they are ok with it.

    • nomel

      yesterday at 11:09 PM

      How is that in any way related to a consumer device? This method doesn't reduce physical memory requirements, so still results in huge die area. This isn't a for-end-user thing, probably for decades.

      • twobitshifter

        yesterday at 11:11 PM

        Ok, how long until nvidia gives us a new GPU?

        • nomel

          yesterday at 11:17 PM

          I don't follow. How is that related? GPUs don't have fixed memory. You don't throw them away when you want to load a new model.

          NVIDIA will probably give us a new GPU when someone competent in the free market decides they want wheelbarrows full of money. Unfortunately, AMD is entirely, incomprehensibly, incompetent, to the point where I can only assume they're colluding with Nvidia, behind the scenes.

• bamboozled

  yesterday at 9:53 PM

  It googles models suck

Thinking that five or six years from now, Fable-level intelligence could be provided at 100x the current speed... makes me feel lost. I cannot imagine what the future will look like.

• ilaksh

  today at 1:18 AM

  Cerebras already runs large models like Kimi 2.6 or GLM at like 30x speed. 100 times is next year, not six years.

  You can actually test it out on their website, just imagine 3 x faster and maybe 15% smarter.

  • kllrnohj

    today at 3:59 AM

    Cerebras is literally the entire wafer, so it can't get bigger. So where is the jump from 30x to 100x coming from? Node improvements only yield like 10-20% gains these days...

    • ilaksh

      today at 4:14 AM

      They have a next generation, I don't really know if it will be 3 x or what but I heard it was significantly better.

      Also there are other people innovating in hardware.

  • keepupnow

    today at 1:42 AM

    This.

• dyzone

  today at 4:07 AM

  It tells me that they have some kind of insider knowledge that the models have hit their limits and won't be getting much better, and it makes sense economically speaking to just bake the current models and use them for the next 5-10 years. Looks like we're near the top of the S curve.

  • hahahaa

    today at 4:47 AM

    A model you can run for practically no cost is a new proposition. It is the CPUification of AI. Sure there be supercomputers but you PC will be pretty super too.

  • mountainriver

    today at 4:22 AM

    What would possibly tell you that?

    • today at 4:48 AM

• __MatrixMan__

  today at 4:41 AM

  Last month: agents spend 4 days on a hack, humans spend 3 weeks (so far) digging through the slop to figure out what happened

  Next time, one of those number will be smaller, and the other will likely be bigger. How long before the analysis side gets too overwhelming to bother with? Probably less than 6 years.

• 1saadcodes

  today at 3:19 AM

  Feels both unreal and dystopian. The speed at which these models are developing is very scary

• DiscourseFan

  today at 1:08 AM

  It will be cool but also violent and terrible.

  • pizzaiolo

    today at 1:22 AM

    So, like the present

    • barbazoo

      today at 1:44 AM

      With more wealth concentrates at the top, yes.

      • __MatrixMan__

        today at 3:17 AM

        Like some kind of pimple, which we can pop from all sides. And then we'll build something different. Something that works.

      • bigyabai

        today at 2:34 AM

        Presumably wealth would concentrate upwards even if AI was never made.

        • sanex

          today at 3:12 AM

          Yes it's a function of the monetary system. Absurd amounts of debt only certain people can access.

The demo: https://chatjimmy.ai/

• walrus01

  yesterday at 9:39 PM

  I know it's a relatively tiny model, but damn, is that thing fast.

  It also mostly passes the "schlong" test

  https://pastes.io/YcxSi8Fp

  • andix

    today at 12:07 AM

    It failed on my usual test. But it failed really fast:

    "A farmer has a wolf, a goat, and a cabbage. The wolf is imaginary and doesn't exist. He wants to cross the river, but the boat is only big enough to hold him and one of them. The farmer can't leave the wolf and the goat together, because the wolf will eat the goat. Similarly, he can't leave the goat and the cabbage together, because the goat will eat the cabbage. What is the smallest number of trips the farmer needs to make to get everything across the river?"

    • tyre

      today at 3:54 AM

      This farmer needs a tote.

  • AussieWog93

    yesterday at 10:01 PM

    I read the paste, it got the etymology wrong, no? Schlong comes from shlang (snake), not shlemp (is this even a word? I don't speak Yiddish but couldn't find it on Google).

    Oxford also claim that its first recorded use was from the 60s, not the 20s; https://www.oed.com/dictionary/schlong_n?tl=true

    • walrus01

      yesterday at 10:17 PM

      It did get it wrong but it also got a lot farther than much more recent, but worse models like 6.7GB on disk size ternary bonsai. It at least knows it's from Yiddish. The "schlemp" appears to be a total hallucination or it's confusing it with schlep, which is not related to schlong. One of the reasons why I said it "mostly" passes the test. Something much larger on the size of qwen 3.5 122B, deepseek v4 flash or similar that runs in 120GB to 190GB of RAM in my experience will answer perfectly unless it has been ruined by something like Q2 quantization.

  • thoughtpeddler

    yesterday at 9:44 PM

    I didn't realize there was a SchlongBench™ (but of course there is). What's it test? (asking seriously)

    • walrus01

      yesterday at 9:47 PM

      There isn't SchlongBench(TM) yet, it's a specific question I've been asking of differently sized models as a randomly chosen gauge of how much less commonly used knowledge is perma-baked into it. In this case a question about a specific yiddish origin slang term. Small/bad models don't know it's from middle high german or Yiddish and get its origin and meaning totally wrong (or it runs into model censorship related to slang related to the male anatomy).

      It's also a question I have found will cause models that don't know what it is to go off quickly in a direction of hallucination trying to explain it, so the hallucination is evident very quickly starting from the first ever prompt issued with 0 context fill. Example: I had a model write four detailed supposedly-accurate sounding, grammatically correct paragraphs saying its origin is from AAVE (African American Vernacular English), which it most certainly is not

      You could do the same by picking any topic that is very rarely discussed in conversation, some esoteric and narrow piece of knowledge and asking the model about it.

      • thoughtpeddler

        yesterday at 10:09 PM

        Oh ya, this is like the approach from the Incompressible Knowledge Probes [0] paper - smart!

        [0] Incompressible Knowledge Probes: Estimating Black-Box LLM Parameter Counts via Factual Capacity [https://arxiv.org/abs/2604.24827]

• wxw

  yesterday at 9:22 PM

  I freakin' love this demo. It feels magical.

  • VBprogrammer

    yesterday at 9:44 PM

    I had the same reaction but then I showed it to my partner. She completely didn't get it, in her words "how can it be thinking of a good answer when it's that quick?"

    I tried to explain but I fear were probably going to be adding artificial sleeps to these things to convince the masses it's doing something clever.

    • varun_ch

      yesterday at 10:06 PM

      to be fair, the model used for Chat Jimmy is not very smart, but the world where it is smart is very interesting.

      It’s going to be really crazy when the bottle neck for agents is the speed of the tool calls rather than the speed of inference. Imagine an agent interacting with the terminal near instantly…

      • ricardobeat

        yesterday at 11:45 PM

        I had the chance to try out MiMo v2.5 Pro Ultraspeed (600-1000tok/s) for a couple weeks and it is amazing.

        Developing software becomes 95% about intent and requirements. Can’t wait for the next iteration of that.

    • axus

      yesterday at 10:12 PM

      I asked it some old hardware command line questions I'd recently asked Gemini, it hallucinated parts of the answer.

      The characters in the 3-act Shakespearean play had very little depth, many of the names were similar, and they were not very smart, but the simple plot was cohesive.

    • XCSme

      yesterday at 11:12 PM

      Wait, is it even thinking? Or is it an instant model?

      • msdz

        today at 3:03 AM

        It’s not reasoning, the hardware demo uses a 3.-something generation Llama 8B.

        But it’s proven they can automate this (they didn’t etch eight billion weights by hand after all, obviously), so now the interesting question is whether they can scale it to more recent aka bigger models.

        After all, there’s already very useful models even for productivity at 27 or 35B.

        • XCSme

          today at 3:52 AM

          My concern is that reasoning could involve some sequential steps that instant models don't.

          Not sure if modern models "think" only by outputting <thinking> blocks, or there is a more complex mechanism at play.

  • emdash

    today at 2:43 AM

    I feel like Ray Kroc in the McDonald's movie trying to figure out how his hamburger could possibly be done when he just ordered it

  • pizzathyme

    yesterday at 11:27 PM

    For those old enough to remember, this is like dial up internet to broadband. So fast it creates new markets

• ecshafer

  yesterday at 11:46 PM

  That is insanely fast. I had it generate a basic C FFT library that can handle multi-dimension arrays, and it was instant.

• senderista

  yesterday at 9:42 PM

  Wow, feels like Google web search in 1999.

  • joshvm

    yesterday at 10:16 PM

    If you still want the experience, go and browse McMaster Carr. Wizards designed that website.

    • senderista

      today at 2:33 AM

      Oh I have, though not for a while.

  • jodrellblank

    today at 3:15 AM

    or LiveGrep fast search of the Linux kernel source code with regex support: https://livegrep.com/search/linux

• brikym

  yesterday at 11:34 PM

  The speed is awesome, in the true sense of the word. It's great at knowledge and basic stuff but the output is complete junk for anything concerning new facts or slightly esoteric topics.

• appplication

  today at 12:10 AM

  This is the coolest LLM thing I’ve seen since the original ChatGPT announcement a few years ago. IMO much more impressive than marginal gains of frontier models.

• hahahaa

  today at 4:51 AM

  Made me an entire app in 84ms lol

• mintflow

  today at 1:41 AM

  try let it to get a brief of france history which being reading a while hit the button and then the brieft jump into my eye

  Generated in 0.051s • 14,092 tok/s

  Impressive...

  Given gpt 5.5 was very good to me and gpt 5.6 series seems not boost too much, i kinda like the way bake the model weight to the chip, and connect multiple chip to serve the large scale model and allow respin some parts(ROM like?) to do model weight update, maybe this seems sustainable, the future is exciting

• XCSme

  yesterday at 11:11 PM

  Wow, that's instant, crazy.

• itvision

  yesterday at 9:18 PM

  OMFG this thing is fast.

  • phoh

    yesterday at 9:58 PM

    its fast but try to get it to give you pi to 50 decimal places. it didnt go well for me.

    • walrus01

      yesterday at 10:01 PM

      I think the same exact model running on CPU-only and RAM, or a small GPU, would do about the same? It's quite an old model now and small, you could throw a GGUF into llama-server or something for a side by side comparison.

      https://huggingface.co/meta-llama/Llama-3.1-8B

      As I remember just about any english language model from mid 2024 and earlier didn't even do well if you asked it to count sequentially from 0 to 100, nevermind calculating stuff.

    • estearum

      today at 12:04 AM

      That's not how LLMs work

• mrheosuper

  today at 1:48 AM

  looklike the training material is stopped at around July 2022, a little too outdated.

• hendurhance

  yesterday at 9:40 PM

  I understand the appeal due to the speed

• yesterday at 9:58 PM

• anigbrowl

  yesterday at 10:56 PM

  15,000 tok/s

  ....damn. It's very impressive notwithstanding its limitations.

• zhoge

  today at 1:00 AM

  This is the answer I got after asking it twice what's taalas (second time hinting that it's a chip startup):

  After a quick search, I found that Ta'ala is actually a Canadian chip startup that produces artisanal, high-end potato chips. They offer a range of unique and creative flavor combinations, often featuring Canadian and international ingredients.

  Ta'ala is known for its high-quality, small-batch potato chips made with premium ingredients and care. The company is committed to creating unique and delicious flavor profiles that showcase the best of Canadian ingredients and cuisine.

  Is this the Ta'ala you were thinking of?

• nsxwolf

  yesterday at 9:12 PM

  It doesn’t believe it’s running on that chip, it’s arguing with me

  • shaewest

    yesterday at 9:20 PM

    It's running a very small, non-reasoning model at the moment. But more generally, almost all LLMs argue on the hardware/model they are/are on.

    • metadat

      yesterday at 9:36 PM

      What would tokens/sec performance look like for a reasoning model? An order of magnitude slower?

      • penagwin

        yesterday at 9:45 PM

        Reasoning models are the same speed. They’re just post trained with RL to do CoT inside tags like <thinking></thinking> before a tag like <response></response>

        There’s no difference in the inference implementation, parameter count, or speed.

        • paytonjjones

          today at 1:23 AM

          There's a difference in the latency distribution between when you submit a query and you see the response, which is what the comment is (clumsily) asking about.

          But yeah, there are a lot of factors, so it's hard to answer, and tokens/s isn't the right question.

    • dumberquestions

      yesterday at 9:23 PM

      Which model? Or how many active parameters?

      • _whiteCaps_

        yesterday at 9:27 PM

        Llama 3.1 8B model

        • dumberquestions

          yesterday at 9:48 PM

          So this demo is around 90 times faster than typical speeds for the same model at openrouter, and around 30 times faster than the absolute fastest option available (Groq).

          • mdp2021

            yesterday at 10:00 PM

            https://taalas.com/h-content/uploads/2026/02/graph.png

            • Gander5739

              yesterday at 10:42 PM

              https://xkcd.com/1162/

          • anthonypasq

            yesterday at 9:58 PM

            im assuming energy expenditure is substantially lower as well

      • yesterday at 9:27 PM

  • wmf

    yesterday at 10:36 PM

    AIs don't intrinsically know anything about themselves so they often give wrong answers to such questions. This can be fixed by putting info in the system prompt but they may consider it a waste of tokens since most usage doesn't benefit from that information.

  • anigbrowl

    yesterday at 11:14 PM

    That proves it's conscious!

    (/s!)

What I like about this, is that it significantly increases the probability of a sci-fi scenario where you're picking up a hot chip on the black market; rumor has it, Mythos 9 weights baked in...

• bigyabai

  today at 2:36 AM

  Plug it in, and it's a old prototype with Gemma 5 weights baked onboard. Dammit, fucked by Craigslist again!

  • NitpickLawyer

    today at 3:27 AM

    Back in the kazaa and limewire days, you'd sometimes try to get a movie / episode from a series, wait hours / days for it to download, and when it was done you had a ~50/50 chance to actually watch what you wanted or an old german porn movie :/

Can anyone imagine if a video generation model with the speed of ASICs baked into silicon ? real Sci-fi

This is neat but IMO a little crazy.

Something I personally haven’t seen much of, in all the discussions of model benchmarks and AI breakthroughs, is a distinction between “peak performance” and “reliable performance”. The “peak performance” of frontier models is very high: they’re solving open math problems, analyzing large codebases, etc. But my subjective impression is that “reliable performance” is mid at best: out of 100 random questions I might think to ask, it’s likely to say something wrong or stupid a handful of times at least.

I think there’s inherent tension between the two: the more a model reaches or outright hallucinates, the more likely it is to come up with tricky, subtle solutions to problems (I think people are somewhat like this too: Terry Tao’s brother is nonverbal, Jim Watson’s son has severe schizophrenia, etc). But then the less likely it is to generate a sensible email reply.

I use models all the time for coding, but I would not let one take over my daily correspondence. If the idea here is to run frontier models at high speed in data centers, that could be useful (the speed would be cool), but I’d be surprised if the cost of that hardware churn is worth it to frontier labs. But if the idea is to turn this into a chip that goes in your phone as some kind of routine, low-power inference thing…taking something too kooky to be relied on and baking it into your phone’s hardware like that doesn’t make sense to me.

• dumberquestions

  yesterday at 11:16 PM

  I think you're underestimating both their reliability for standard problems and the usefulness of that level of reliability.

  • tyre

    today at 3:58 AM

    This is a good point. Opus does some silly shenanigans sometimes but then catches it later. It’s still an order of magnitude faster at getting to a working system than I am, for ones I don’t know.

    It’s really a dream for setting up a homelab

• daishi55

  yesterday at 10:56 PM

  > out of 100 random questions I might think to ask, it’s likely to say something wrong or stupid a handful of times at least.

  What are some examples?

  • wmf

    yesterday at 11:25 PM

    There's a benchmark for this and a lot of models get negative scores because they're so unreliable: https://artificialanalysis.ai/evaluations/omniscience

    • daishi55

      today at 1:26 AM

      I wanted some examples they actually experienced. Because I use these things daily and haven’t seen a hallucination in a long long time.

      • yoyohello13

        today at 1:42 AM

        I saw a hallucination just this afternoon about a spurious ca cert error. Definitely happens less often, but I do need to correct it occasionally. Maybe once a week so it still requires vigilance.

      • tyre

        today at 3:59 AM

        Search a terminal with Claude Code for things like, “I got it wrong twice. I should look up the documentation instead of guessing.”

        Does it about once a day, that I notice.

Given the fast churn of the models, how does it work out?

Won’t the silicon etched model already be 1 or more versions behind by the time the silicon comes out.

Though if it’s cheap enough, there certainly can be a market for cheaper model inferences.

• hahahaa

  today at 4:53 AM

  They still make 6502s right.

• sigmoid10

  yesterday at 10:31 PM

  I find speed alone would be a game changer for current models. I hardly find any task anymore that the current frontier models can't do with max reasoning after several rounds of feedback (provided sufficient instruction and the right harness). But waiting an hour or more for reasoning to finish is getting really cumbersome. If they could do the same in seconds (and for cheap of course), I'm pretty sure we'd pretty soon see major software companies pop up that are run by a single human.

  • deadbabe

    yesterday at 11:15 PM

    Can you give some examples of these tasks that require an hour or more of reasoning?

    • xyzsparetimexyz

      yesterday at 11:36 PM

      The recent maths prompts did. The 'you should find a breakthrough' one was several blocks of reasoning, each taking 90 minutes or so

• craftkiller

  yesterday at 11:53 PM

  I think the real value here is not as a customer-facing agent/chatbot but for for automated processes. Think of all the companies out there that have LLMs doing simple tasks like categorizing customer feedback emails. For such tasks, you don't gain much from better models, so if you could run it 10x cheaper on a slightly older model, it would absolutely be worth it. Pretty much any place people are currently running a flash model could benefit from this since they're already deciding that speed+price is worth using a less capable model.

• XCSme

  yesterday at 11:14 PM

  I think this would make sense for consumer hardware, not for AI companies.

  AI companies constantly update/change stuff, new models come out, new requirements, etc.

  But if you ship an "ai-powered" dishwasher, it can come with the chip built-in to do computer vision and precisely target each spot, and will be sold as-is with no updates.

  • yumraj

    today at 12:41 AM

    Makes sense. Actually to expand, I believe this can make a lot of sense for industrial robots and such which have a more or less fixed job and latency matters more, so a well tested model may be more valuable than need to keep updating them

  • throwaway173738

    today at 12:37 AM

    You don’t need this chip to do that. Computer vision has used machine learning for decades. The task you’re describing is pretty rudimentary and an off the shelf model with a control system would do it way cheaper.

    • tyre

      today at 4:01 AM

      Think of a HomePod. 99% (and likely much more) of what people are asking is super simple.

    • XCSme

      today at 12:42 AM

      It was just a random example, you could think of it as being a lot more complex (detect which type of food it is, what detergent to use, how much water, remember patterns, learn over time, adapt, etc.)

  • m463

    yesterday at 11:51 PM

    subscription "ai-powered" dishwasher with personalized user ads, most of the chip dedicated to "personalized" not spots.

    • XCSme

      today at 12:25 AM

      So local personalized ads?

      Not sure if that's better or worse than online personalizaed ads...

• christina97

  yesterday at 11:41 PM

  There’s some kind of tradeoff between speed, cost, and quality for every application. I would be perfectly happy with a model 6 months old that was 50x faster for many uses. Right now I use either Opus (for smart stuff) or Flash without thinking (for fast stuff). I would take an even dumber model for more speed (lower latency in particular).

• nullbio

  today at 1:04 AM

  Perfect for consumers. You buy it and then you need to buy a new one in a couple of years. If they can make them affordable they'll sell like hotcakes.

  • etoxin

    today at 1:45 AM

    And the second hand market. I'd love to see this integrated into motherboards like RAM. Someone could have a motherboard with 4 sticks of different AI with various models. Swap, change and trade.

• chorizo

  today at 1:36 AM

  That’s not going to be true forever. As models mature, we will hit diminishing returns. Major improvements will come annually rather monthly - matching the roughly annual release of new processors. Model ROM’s will likely get integrated into die packages just like DRAM now.

  • pennomi

    today at 1:54 AM

    I’m hoping for SNES style cartridges

• mrheosuper

  today at 1:50 AM

  I'm still using Opus for most daily task because Fable is too expensive.

  If they begin etching Fable into silicon now and release it 2-3 years later, i can see the market for it

• noosphr

  today at 2:11 AM

  This is a feature for most local use cases. You don't want all your work flows to start failing because of a model update.

• brokencode

  yesterday at 11:27 PM

  Already models have gotten really good at a lot of things.

  A lot of people would probably be happy to stick with the same model for a year or two if it’s 10x faster and cheaper.

  And perhaps older models can become cheaper over time as newer models come out on new silicon for a higher price. That incentivizes people to stick with older models.

• prinny_

  yesterday at 10:55 PM

  They expect a sort of breakpoint at which each subsequent model version will only be marginally better than the previous ones, thus allowing them to retain their value for some time. Their business doesn’t work if each year the new model demolishes the previous one in terms of performance.

• laweijfmvo

  yesterday at 11:16 PM

  pretty much everything is “1 or more versions behind” by the time it comes out. the question is whether or not it’s still useful? at some point, presumably not every application will need the latest cutting edge huge model.

• casey2

  today at 2:11 AM

  There isn't a fast churn in the underlying pretrained model, nor RL. It's mostly orchestration around the model. Said another way you could just pretrain and RL for longer.

  Also I believe there is both a market for extremely fast local inference with current model performance and that such fast inference would unlock unforeseen usecases. Especially as TPS approaches early computer clock cycles and data rates.

• deadbabe

  yesterday at 11:14 PM

  You could take your silicon chip and have it re-etched only with model diffs for an upgraded version.

  • yumraj

    today at 12:37 AM

    How does that work, as in re-etching of silicon? Any pointers to read?

This is probably a win-win. The team gets paid, and we get greater assurance that their best ideas and architectures -- which are truly impressive -- are going to see the light of day in actual products.

• badatnames

  yesterday at 8:55 PM

  They were too small for this to be a meaningfully sized purchase for AMD, there's real risk they get sucked into a team that ultimately delivers sqat, not to mention the chances of anything being delivered in an even remotely consumer-priced bracket are definitely out the window

I've been eagerly awaiting their 2nd gen HC2, which uses multiple chips to host a "mid sized reasoning" [1] model. Its due in summer according to the article, I wonder if it will ever be released in that form now.

[1] https://www.forbes.com/sites/karlfreund/2026/02/19/taalas-la...

• NitpickLawyer

  today at 4:02 AM

  > I wonder if it will ever be released in that form now.

  Yeah, I had the same thought. The key thing for them was the price point at which they could deliver a ~30B model. I would buy one today if it was ~1000$ and could run whatever the best 30B model is today, at those speeds advertised. Even if the model becomes superseded by model.5 in a few months, there's still a lot of things you can do with a "good enough" model for some tasks. And things like maj@x or generate 10 times and choose "at a glance" what you like (think frontend stuff) would be worth it.

  No idea if them selling to AMD is good or bad.

Question: we currently emulate neural networks by performing matrix math in synchronous clock CPU architectures. Would it not be better to abandon synchronization and etch neuron synapses directly in silicon, keeping only the weights variable? I think some researchers are pursuing this, but I forget what the approach is called.

• freakynit

  today at 3:37 AM

  "Neuromorphic chips" .... and I have the exact same question in mind.

Well so much for that dream.

Guess we can look forward to picking these up ex-enterprise on ebay for under $5k a pop in a decade or two

• dogomatic

  today at 12:51 AM

  What’s stopping competition?

Is there any LLM from exactly one year ago that would be worth running?

In Aug 2025 you had

- OpenAI o3

- Opus 4.1

- Gemini 2.5 Pro

- Grok 4

Even if those were almost free to run, you'd be way better off with Deepseek flash 0731 or GPT 5.6 Luna, which already are almost free.

Other than for things where the t/s are critical, it seems like a bad idea to etch a model into silicon.

• mdp2021

  yesterday at 11:04 PM

  > Is there any LLM from exactly one year ago that would be worth running?

  Bad perspective: consider the correction: "when are thresholds of sought quality reached"? Hence: not "is there a 10yo from last year that could compete with the current 13yo", but "will there be a 30(?)yo from last year that could compete with the current 33(?)yo" ('(?)': the scale of yearly growth in the future is uncertain).

  • redox99

    yesterday at 11:17 PM

    It's not just about it "being smart enough". It's about there being actual user demand when it needs to compete with the shiny new model.

    A 10 year old iPhone is probably good enough, but is there demand for it? In a vacuum a 10 year old iPhone is good, but why would you pick it if you can have a current one for a reasonable price?

• singingtoday

  today at 2:14 AM

  We still run GPT 4.1 for some of our use cases. We want to replace it but are having trouble finding models that are as fast with similar or better intelligence.

• daishi55

  yesterday at 10:57 PM

  That is fkin wild. o3 was just a year ago? The progress is truly insane.

  • redox99

    yesterday at 11:08 PM

    Yeah I had to double check, o3 feels like it was ages ago. But GPT 5 came out Aug 7, so it's only one day off from my 1 year ago cutoff!

Waiting for intelligence on a stick, plugin an USB, characters in, characters out.

100% local and no leaks.

AMD could have saved their money and used their own hardware! I've got a language model doing 60k tok/s on AMD hardware already, a Xilinx Kria K26 SOM, with the weights baked into URAM/BRAM with zero DRAM in the token loop. Same thesis as Taalas: single-stream decode is bandwidth bound, so stop fetching weights from far away.

Caveats stacked high, obviously. It's 3.16M parameters (tinystories, and I also have a kevin-speak lemmatised version), the tokens are characters, and the 60k record is 16 streams that each remember exactly one token of context, so it's blisteringly fast at saying nothing. The honest build with full context and KV caching still does ~19k tok/s on one stream though.

I keep messing with the blogpost with the live demo, but I'm planning on flipping it to live in the next day or two

• Melatonic

  yesterday at 10:39 PM

  Yeah Im surprised nobody is talking about this. When everyone first saw Taalas I looked at the design and it had a big legup in physical cache availale compared to most chips. Makes you wonder how much of a benefit there is to the actual "baking" of the model vs just having a large chip with a ton of SRAM (or whatever) soldered close to the edge physically.

  I feel like what we really need is the ability to solder computer cache on all sides of the chip Meaning above and below as well. If you can only attach it to the edges you will be inherently physically limited on the amount you can put (and maybe even have latency benefits as well)

  • Legend2440

    yesterday at 11:00 PM

    What you're describing is what Cerberas does.

    Talaas is different, it's a true compute-in-memory architecture where the weights are stored in the connections between the transistors that perform the matrix multiply, rather than in seperate memory cells.

    Most of the benefit comes from this architecture; hardwiring the weights into the silicon is just the easiest way to implement it. SRAM requires too many transistors, DRAM requires an incompatible manufacturing process, and exotic phase-change memories aren't readily available.

    • Melatonic

      yesterday at 11:16 PM

      Cerebras chips are massive and do have more on the edge but they dont have any top or bottom cache do they?

      • jaggederest

        yesterday at 11:43 PM

        They can't due to power density, I believe - they have to be run in a sandwiched waterblock with massive cooling, as far as I can tell. That's the biggest thing that baked weights gets you - a relatively modest watts-per-square-mm compare to cerebras, where they had to engineer a whole system to get the watts out of the chip

        • pstuart

          today at 4:31 AM

          Do you think there's room for reducing power requirements? Obviously shrinking the process is a win, but is the existing implementation a "just make it work" phase that has opportunities to increase computational efficiency?

          With that kind of speed and if even lower power requirements, they could release mini compute units with USB4/Thunderbolt for plug and play inference.

  • wmf

    yesterday at 11:29 PM

    Taalas does not have cache so...

    I agree that Groq with multilayer hybrid bonding could be a good idea.

• tandr

  yesterday at 9:29 PM

  Well, technically it is their hardware now...

  • questionableans

    yesterday at 9:43 PM

    And their team, if they treat them well.

• zxspectrum1982

  yesterday at 10:53 PM

  1. How come you didn't make your implementation public? You could be a millionaire now. 2. Especially if AMD has the technology to do what Taalas does, it makes a ton of sense for AMD to acquire Taalas: remove them from the market. Make sure nobody else (Intel, Huawei, Alibaba, NVIDIA, etc) acquires them. It could have been a great acquisition for a rebirth of BlackBerry btw.

Smart move by AMD. Chatjimmy is very fast and not very good, but I think it might become very fast AND very good.

I guess the idea is, gains from inference speed could offset the cost of upgrading the chips to a new model when really required. I think general purpose models would consolidate and release frequency might flatten out, favoring this strategy.

Really hoped to see their hw out in the wild one day

Field reprogrammable, it's an FPGA on steroids. Field upgradable.

Burnt in, it needs a zif socket and easy access in every car, aircraft, a pull out slot in a phone, or it's new era planned obselescence.

• XCSme

  yesterday at 11:17 PM

  Why not have some a device/hardware that programs itself on-boot.

  Sort of a FPGA, that (electrically) arranges the connections on-boot, and then it's like a static inference chip.

  • wmf

    today at 12:14 AM

    FPGAs already configure themselves on boot.

    • XCSme

      today at 4:02 AM

      I asked a LLM after posting my comment, to see if I had a genius idea or not,just for it to tell me the same as you, that's now they work already...

• mdp2021

  yesterday at 10:39 PM

  Can that be done when the whole idea is to store a multiplier into a handful of transistors?

  • ggm

    yesterday at 11:03 PM

    I have no idea. It makes my comment a statement posted as a proxy for a question, a question you correctly pose explicitly.

    If it can, then deployment in a sea of gates can make a chip viable across model generations as weights change, inside some scale factor.

    If not, unless the part is under a pinout and address model which can scale on the bus, and can be easily replaced, it makes the entire dependency a replacement, not just this part. So embedded use has consequences.

• xyzsparetimexyz

  yesterday at 11:38 PM

  It can just be pcie

It would be quite ironic if this technology would render all those AI data centers practically useless. If the next step are just a much smaller amount of expensive chips, and the bottleneck becomes manufacturing those chips fast. Not building huge data centers and fighting for electrical power.

I have used chatjimmy before, it is incredibly fast, waiting for latest SOTA model on the chips in future. Great!

> Once the chips are deployed you’re stuck with that model.

At least we can be sure that's the model we wanted. Service providers could be serving modified versions and nobody would ever know.

Toronto Canada startup btw.

• cmrdporcupine

  yesterday at 9:20 PM

  Seems to be somehow some kind of offshoot from or connected to Tenstorrent, which is just down the road. Founder looks like he was/is maybe at Tenstorrent and previously associated with Keller?

  Always fantasize about applying at Tenstorrent, but wrong side of Toronto. 2 hour commute.

• kridsdale1

  yesterday at 9:24 PM

  Works well, I remember driving by the ATI building as a kid.

Wow, we're heading back to mask-programmed ROMs. I'm feeling young again.

People are missing the point if they think this is useless because frontier models keep changing every few months.

We really, really need better secondary models that can do things fast and do them cheaply for lots of dumb tasks. Not only because it can be used as sub agents by frontier models, but also because it can be like a universal grease for all kinds of software.

I've got an app I am building and I don't want to tie myself with frontier models because I'll never be able to beat openai/anthropic. I just want a simple, cheap, instantaneous model that can just go through my documentation and tell the user what to do next and how to integrate with whatever ai subscription they have.

While this design is self-limiting I think its a good approach. It doesn't take an entirely new architecture or infinite memory to produce significant performance improvement.

• Legend2440

  yesterday at 11:03 PM

  This is a new architecture. It's a non-vonn neumann device.

I would've hoped the company stayed independent instead of being engulfed into a behemoth. I'd like to see more diversity in the hardware ecosystem, but I guess the economics of hardware manufacturing aren't there.

Can anyone comment on the economics and likely turnaround times of this process, when it’s more mature?

Would it be realistic for a frontier lab to deploy this or would the turnaround time mean the model is always too out of date?

Assuming the weights and architecture are eventually stable, how much cheaper would this end up being?

• 2001zhaozhao

  yesterday at 9:49 PM

  There are always uses for outdated models.

  Claude Code is still using haiku 4.5 from ages ago for explore subagents for instance. Not to mention production uses like customer service that only need to be "good enough"

  • edot

    yesterday at 9:56 PM

    Just looked this up, no longer true. Explore subagents inherit whatever model the parent is. And you can of course make other subagent configs.

    • samtheprogram

      yesterday at 10:04 PM

      That's solely so that you burn more money. It's totally unnecessary to assume the parent model. Sure, it could be upgraded from Haiku if there was a solid reason to, but...

    • AussieWog93

      yesterday at 10:06 PM

      I mean, if you could get Opus or even Sonnet 4.5 at 1000+ tok/s exploring the codebase, they would probably change that setting back.

      But either way, I think GP's overall sentiment of "delegating intelligence-saturated tasks to an outdated but fast subagent" makes a lot of sense.

  • alightsoul

    yesterday at 10:12 PM

    Customer service has really degraded huh. 4 years ago they expected opus performance out of human call center agents

    I guess losing some customers due to poor customer service is ok if the price of customer service is right.

• cogman10

  yesterday at 10:09 PM

  2 to 3 months optimistically assuming everything goes smoothly and is fully automated.

  6 months or even a year if something goes wrong in the fabrication process and you need to update things.

  If they do more standard asic design, it could be a lot longer as the design needs to be validated on an FPGA cluster, which would necessarily need to be very big for something like a LLM. Easily up to 2 years.

  There's a reason chatjimmy isn't demonstrating newer models and why they only show of an 8B model.

• shangofox

  yesterday at 9:50 PM

  I mean even if it take a few months, it'll still be out of date. But there was a hypothetical when it came up in Feb, would you want Qwen 3.5 at like 10k tokens per second.

  At the time people were no doubt saying yes but now 3.8 is out, is that still desirable?

  • xienze

    yesterday at 10:06 PM

    There's soooo much stuff that such a model is still capable of doing in the pursuit of getting a better overall answer. Imagine a powerful research agent that blasts out dozens of the small, cheap models to fetch and summarize one page each. Then the beefy researcher model performs the final analysis.

They moved from HBM to dedicated silicon and only got a 48x speed up? That is so, so, so much less than I would've expected. Any numbers on how it scales?

What does it take to go from here to a model on a pcie card or an m.2 card, so I can plug one into my workstation / laptop? Will 'intelligence' become much like a gpu, where most people just live with the performance of whatever they have installed, outside large companies that must have cutting edge, or prosumers that have a incrementally better version than the masses?

Are we a couple years away, a decade away, or something else?

• mdp2021

  today at 12:07 AM

  > What does it take to go from here to a model on a pcie card or an m.2 card

  It is already that.

  > Will "intelligence" become much like a gpu

  As an option among the implementations.

  > Are we a couple years away

  They could mass produce now, but it makes no sense at this rate of improvements in the models.

I've been showing people chatjimmy for months - it's incredible. Both reasoning and tool use generation scale with TPS. Imagine 100x more reasoning on a model, or 100x parallel tool uses.

Honestly, this is starting to make more and more sense. SOTA models are starting to converge to certain architecture and capabilities. I wouldn’t be surprised we end up with a base model ASIC + “fine tune” card where it’s a physical LoRA style adapter.

• encyclopedism

  yesterday at 9:17 PM

  Imagine a multi-modal model with 1000's of tokens per second. Realtime inference for a host of applications. This is a BIG deal and will change the landscape in unfathomable ways.

  The https://chatjimmy.ai demo was impressive.

  Once models settle down this makes sense. Imagine a cartridge with a physical model on it. You purchase a cartridge and stick it in your computer/phone/server. Want to upgrade? By a new 'cartridge'.

  This should bring inference cost down dramatically, I wonder how OpenAI/Anthropic feel about that.

  • 2001zhaozhao

    yesterday at 9:52 PM

    i'm looking forward to Qwen3.8 27B launch to see how much models have peaked at a given size.

    it might already be time to start burning the best small models onto hardware since it's possible they can't get much better at many tasks like knowledge recall due to the inherent information density limits for models at a given size.

  • Grosvenor

    yesterday at 9:42 PM

    > Imagine a cartridge with a physical model on it.

    I can finally have my own Dixie flatline. Cool.

    • mdp2021

      yesterday at 10:14 PM

      > Dixie Flatline

      In case some did not know: also the movie (actually TV series) is finally happening.

      # Neuromancer - Official Teaser ( https://news.ycombinator.com/item?id=49055037 )

  • anthonypasq

    yesterday at 9:59 PM

    very interesting idea. i didnt think of that. i was just assuming youd have an additional one of these in your phone for actual lightning fast local inference

• VladVladikoff

  yesterday at 9:14 PM

  Wouldn't this mean someone with sufficient hardware could lift the SOTA model weights off the chip? Or are you saying that these chips would only be used internally by these companies and not sold to the public?

  • dumberquestions

    yesterday at 9:28 PM

    I wouldn't expect companies not sharing their weights today to be any more likely to share them if they're on hardware, this doesn't sufficiently hide weights from a local user.

  • snek_case

    yesterday at 9:21 PM

    The weights are very unlikely to be on the chip itself. That wouldn't work for SOTA models that are terabyte scale, even quantized. This is probably an accelerator for specific kernels in the model, but the weights are likely loaded from memory. The chip may have SRAM to store some of the weights temporarily during inference.

    • foltik

      yesterday at 10:15 PM

      At least in the case of Taalas the weights are physically encoded directly on the chip.

      It’s composed of 4-bit multiplier cells that compute all 16 possible results in parallel. The top metal wiring layer physically selects the one that corresponds to a multiplication with that cell’s constant weight, and routes it to the next layer.

      • mdp2021

        yesterday at 11:56 PM

        Are you sure? Source? (does not seem to be https://taalas.com/the-path-to-ubiquitous-ai/ , for example)

  • syntaxing

    yesterday at 9:16 PM

    I don’t get why this is an issue? You can run Claude/OpenAI SOTA models through Amazon bedrock. These weights have to live somewhere to run on Bedrock.

    • wmf

      yesterday at 10:40 PM

      somewhere = an AWS data center with multiple layers of security and NDAs

      They won't sell/rent/license the weights to an end user at any price because they don't trust your security.

      • syntaxing

        today at 3:44 AM

        I work in embedded space. Just because it’s in hardware doesn’t mean you can’t “protect” it. Most modern software (regardless if it’s hardware or not) can be cryptophically signed.

        • bluezly

          today at 4:38 AM

          [dead]

  • amazingamazing

    yesterday at 9:21 PM

    One idea would be to use an open model.

• kevin_thibedeau

  yesterday at 9:32 PM

  Then we can have machine psychologists pull cards when they run amok.

  • all2

    yesterday at 10:46 PM

    You have a robot. You need it to be smarter. You buy a new model cartridge (probably a PCIE 9.x). Now you need some domain specific skills. You'd like it to be able to cook, and you'd like it to not dent your walls anymore. You buy 'improved spatial reasoning LORA' card and 'Gordon Ramsey's Chef ULTRA9000' card.

    Now your robot can respond sarcastically when you ask for chicken nuggets. Again. It also doesn't dent your walls anymore.

• walrus01

  yesterday at 9:43 PM

  Having a base model ASIC as a physical piece of hardware makes me think of the early days of microcomputer desktop stuff where having a socketed ROM or PROM was a key piece of hardware, and people actually knew/cared what ROM was on their system's motherboard.

  Imagine if like instead of having a specific Mac Plus ROM, you had a thing that looks like a fat ASIC that can hold models sitting on a slotted daughtercard directly next to the CPU and RAM.

• breadislove

  yesterday at 10:04 PM

  we have not converged at all, if you look at how different the chinese models in terms of architecture you can guess that the labs are experimenting a lot as well. we are seeing all different types of hybrid architectures, different attention methods and so on. Of course on a high level its still a transformer but if you take a proper look we are seeing more divergence then a convergence.

• smokel

  yesterday at 9:05 PM

  The technical aspects of SOTA models are not publicly documented. How do you know if something is converging?

  • syntaxing

    yesterday at 9:14 PM

    SOTA American models are not. SOTA Chinese models are. From a physics aspect, closed source models cannot be too far from open source ones in terms of size. There’s only so much you can squeeze out a B100 style cluster even with fancy Dflash style diffusion model for the speculative model.

  • _aavaa_

    yesterday at 9:08 PM

    If we had deepseek v4 flash 0731 etched on a chip it would be more than capable enough and fast enough for so many people's needs, even hardcore engineer.

    • nurumaik

      yesterday at 9:19 PM

      Will be capable and fast enough for 2-3 weeks until new sota drops

      • amazingamazing

        yesterday at 9:22 PM

        If it is capable today why would a new model change this?

        • thombles

          yesterday at 10:11 PM

          I think it’s tongue in cheek. When I first got access to Sonnet 4.5 I remember thinking to myself “y’know if they never got any better and I just had access to this forever then that would be pretty okay”. Turns out my expectations have changed since then and I would like a higher baseline now.

          • singingtoday

            today at 2:22 AM

            Interesting. I've yet to find a model I consider sufficiently intelligent.

            Fable is nice, but still requires a lot of guidance for large scope tasks.

        • catchnear4321

          yesterday at 9:33 PM

          if capability is a commodity then the differentiator becomes taste.

        • FridgeSeal

          yesterday at 9:32 PM

          Because new stuff instantly makes anything prior bad and incapable and garbage of course! Did you forget the hype-machine speaking notes??? /s

  • cyanydeez

    yesterday at 9:08 PM

    if they were still exponentially increasing, they wouldn't be preparing for an IPO. IPO is where companies go to die and founders escape.

• cyanydeez

  yesterday at 9:10 PM

  I don't think there'll be a fine tune card; you'll have the base model vintage whatever year, and then your GPU will do whatever LoRA layers you want it to do; the LoRA will wrangle older dated models into the current of whatever your looking at.

  But yeah, for things like programming, if it can do linux and python and some go and sql and javascript, larger domains can be threaded with LORA

I think the big news is that AMD is getting into memory-business so they won't be so dependent on Hynix and what have you. Memory is the bottleneck currently.

It obviously won’t be continuous delivery but could make sense if the lifecycle of a model (train, deploy, iterate (meaningfully) is about 1-2 years. In that case it fits nicely in the “this year’s model” already established with cars, phones, etc.

Didn't even give them a chance to launch the hardware.

I’ve been using Gemma as my default (via Kagi) because it’s served on Cerebas hardware. The speed is honestly a game changer for day to day queries.

Can you imagine in few years getting Fable level intelligence at 20k tokens per second?

"You are not prepared" --Illidan Stormrage

So, Kimi K3 in silicon sometime soon?

How's that jive with the fact that they're introducing a new model every other week?

• drchickensalad

  yesterday at 11:40 PM

  The new model every week is not necessary at this point really. What if you could run opus 5 for the next couple years at 1/20 the cost?

  • roughly

    today at 12:23 AM

    What's interesting about this is that I as a user would find this useful, but I think the AI industry as a whole would find it an absolute goddamn disaster. Opus 5 is a very good tool, but it is not a human-replacement-level intelligence, which means the entire revenue stream the industry's built on - labor replacement - is not met by this, and the only slightly charitable read of the industry's finances is that they're gonna bootstrap their way to creating the labor replacement hypothesis by getting people to spend money on Opus/etc, whereas if the actual product is a 1/20th the cost Opus-on-a-chip, the entire business and financing model that's tying up $N Trillion dollars of investment money goes out the window.

    Great for us, looks like a recession as far as the Market is concerned.

• jaggederest

  yesterday at 11:41 PM

  Pipeline the burn into silicon, lower the latency as much as you can, for the 10-100x operation cost it's worth it. Imagine if frontier models cost $5/mtok and the 2nd or 3rd tier models cost $5/billion tokens for 3-month-old models.

With web search and tool call a decent current generation model at the speed of the chatjimmy could do a lot. People saying it would be out of date are missing the point. It’s not going to make much sense for frontier companies that’s chasing the SOTA. But for a lot of business use cases if someone can put GLM 5.2 and sell it as a box, it would make so much sense.

My partner has been asking for a “completely private” model for doing research and shifting through volumes of data that can’t leave the office and $$$ for the current hardware makes no sense. It would be an easy sell if someone walks in with a black box that contains “ChatGPT”.

• 5555watch

  yesterday at 11:23 PM

  In my understanding the first Deep Think / Pro models were already very good as they were doing some kind of parallel repeated reasoning, thus were slow and expensive. So if chatjimmy speeds enables a fast deep think level performance, I think that would be great.

• equinumerous

  yesterday at 10:05 PM

  100% agree - you don't need the most up-to-date model to have something that's useful in agentic contexts. They could even produce chips with weights that make all the decision making/logical reasoning and have it delegate to other specialized agents. If it becomes cheap enough to print a run of custom chips, releasing a batch for each major advancement does not seem unreasonable for SOTA companies.

• cephei

  yesterday at 10:12 PM

  There are so many use cases for supremely fast offline models. The first thing that comes to my mind is for real-time video processing or other non-textual content in real time.

  • anigbrowl

    today at 12:53 AM

    I wouldn't call it supremely fast but zippy and versatile, yes: https://shop.m5stack.com/products/ai-pyramid-computing-box-p...

If things like this get traction, will we need all the datacenters?

• downrightmike

  today at 1:42 AM

  You are mistaken about what the datacenters are for

Core rope memory is back baby!

Enjoying the Ian Cutress / TechTechPotato video on Taalas. Some ok good technical details on the tech, and some good insider baseball, whose who stuff. (What a treasure having tech discussions like this about.) https://youtu.be/3MKRjt59hh4

Congrats to the Taalas gang.

See also: Twitter statement from Taalas https://x.com/taalas_inc/status/2085458427757937097

A massive L for Canada

I feel like NAND process tech could become useful at solving some of these problems. A GPU where you can update the weights a few thousand times may be sufficient.

• mdp2021

  yesterday at 9:41 PM

  The basis of Taalas is "compute in memory" electronics - past Von Neumann's separation of processor and memory.

  You need to be able to add|mul where the data (the weights) are stored.

• addaon

  yesterday at 9:33 PM

  NAND hasn't been scaling great lately. It seems like PCM or MRAM would both be better fits.

• kridsdale1

  yesterday at 9:22 PM

  FPGA model storage?

"... the chip serve Meta’s Llama 3.1 8B at a blistering 16,960 tokens a second — when announced last February, that was 48x faster than Nvidia's GPUs and 8.5x faster than Cerebras' accelerators. "

Token quantity will have a quality all its own.

so qwen3.x-27b on hardware? or better deepseek-v4-flash on hardware .

• ilaksh

  yesterday at 8:59 PM

  I wrote them an email asking for PrismML Bonsai 27b Ternary which is like 6b or something crazy small and would be a lot easier for them to do initially.

  • mdp2021

    yesterday at 9:55 PM

    They were specializing their forthcoming system on 4-bit FP - which I understand is a structural decision.

    Bonsai Ternary (1.7bits/weight) is a compromise, compromise that has to make sense in the context - efficient when translated into transistors.

It must be a “super model”. What will be if new model released? New chips?

• downrightmike

  today at 1:44 AM

  Chip pops out like a gameboy cartridge. AI not working? Blow on it and jam it back in

taalas is great for llama 3.x 8B models, really bad for one board serving Kimi K3, it seems like you would bottleneck at a few hundred tokens no matter what you do.... spreading the big model against multiple cards seems the only way to get into the 1k+ tok/sec range. Another thing taalas is doing is masking the model weights into the silicon itself, not a flashable firmware, which would increase latency....

I've had an endgame idea in mind for a while.

Models, probably first open weight ones like Kimi K3 class, are etched into silicon like this and sold as cartridges almost like old school game cartridges.

You buy a USB-C dongle that the cartridge goes into, or for data centers you have PCI cards that take these in slots.

• wmf

  yesterday at 11:26 PM

  Each cartridge costs $1,000. Do you still want it?

  • anigbrowl

    today at 1:00 AM

    For fast Kimi K3? You're damn right I do

    • wmf

      today at 2:26 AM

      $1,000 only gets you the Qwen 27B cartridge. For Kimi K3 it would be more like $100,000 (and the "cartridge" is the size of a refrigerator).

  • singingtoday

    today at 2:29 AM

    Yeah. I have 3 max20 plans.

  • api

    yesterday at 11:41 PM

    Me? Probably not. A business or a hoster, sure. There'd probably end up being an aftermarket in used cartridges with slightly older but still good models on them.

Yeah, so https://chatjimmy.ai/ ... the model is crap, but the speed is amazing. Worth checking out.

Imagine the size of chip needed to 'etch' something like Qwen 3.6 27B in size.

• golem14

  yesterday at 9:53 PM

  Interesting thought, because it's a yield question. How tolerant are models today to a few broken weights.

  If tolerant, they could churn out many cheaper chips, some perhaps with slight abnormal tendencies ;)

  • thepasch

    yesterday at 10:05 PM

    > How tolerant are models today to a few broken weights.

    Extremely! You can remove entire layers and the model will still work just fine, with barely perceptible capability losses.

    I've cut/bypassed ~15% of total parameters out of Gemma 4 31B on a pod once. Still got perfectly coherent responses out of it. Certain layers are a lot more important than others, particularly early and late ones; but it's honestly astonishing how much can be cut out from the middle without destroying the model's coherence.

    I didn't run any meaningful benchmarks, so I have no idea what the capability loss looks like exactly. But "produce coherent and sensible English in response to a wide variety of prompts" was definitely not among the things the model unlearned.

    • walrus01

      yesterday at 10:26 PM

      Brings to mind the scene in '2001' where Bowman is pulling out individual pieces of hardware that represent the mind of HAL, and it becomes increasingly incoherent as more physical hardware is detached.

      https://www.youtube.com/watch?v=UwCFY6pmaYY

  • walrus01

    yesterday at 9:56 PM

    I wonder if you had a few percent of problems in the yield, if it would be functionally equivalent to the difference between a unsloth-published Q6 standard size GGUF vs. the nearly perfect precision of an unsloth Q8-K-XL. Or more like Q4 vs Q8 where a lot is lost.

• mdp2021

  yesterday at 9:46 PM

  Not too dissimilar to the first HC1 (6nm 815mm² 53B Transistors embedding an 8b LLM):

  > Our second model, still based on Taalas’ first-generation silicon platform (HC1), will be a mid-sized reasoning LLM

• flog

  yesterday at 9:45 PM

  If someone has that sort of knowledge; how big a chip would be required? Is it possible?

  • mdp2021

    yesterday at 9:50 PM

    Well, given the data above, roughly a 220b transistors chip for the HC1 tech.

> At 20 billion parameters per chip, you’d need just 50 accelerators to support a trillion-parameter model

I don't see any evidence that this is possible. From my understanding, the whole model needs to be on a single chip. Which rules out any popular frontier models with several trillions of parameters. Even smaller sub-frontier models have hundreds of millions of parameters, so these would be ruled out as well.

• wmf

  yesterday at 10:45 PM

  The methods for splitting weights across multiple chips are well established. Groq/Cerebras can't hold a model on one chip either.

  • pyrolistical

    today at 1:26 AM

    Umm I have an extra 35, do you have layer 6?

• octoberfranklin

  today at 3:31 AM

  They pipeline-parallelize across multiple chips. DeepSeek v4 Pro will be 30 chips.

• IsTom

  yesterday at 10:40 PM

  I think it's enough that a single layer fits on each chip if you can daisy-chain them with good interconnects.

Taalas is just a phenomenal startup from Toronto. My dearest congratulations to the founders.

Edit: Lol, downvotes? Stay jelly, meanwhile Talas goes brrr.

[flagged]

[flagged]