today at 3:28 AM
It's absolutely true. I have three different setups here, a rig with 8 gpus that I use in either one of two modes (all 8, very heavy model or just one much lighter) and two DGX sparks. I haven't used a paid model in months. Sure, they use a few more tokens and the results may be slightly less good (but this doesn't really matter because I'm still in the loop so it is easy to correct). But the fact that I'm not sharing my work with the paid providers is much more to my liking than having big brother looking over my shoulder at every step and occasionally giving me the 'I can't let you do that, Jacques' treatment.
today at 5:17 AM
Gaming used to be done in arcades for 25c a play*.
Over time NeoGeo realized they could sell you the hardware to play at home (āon-premā) at arcade quality albeit at a price out of reach to most people. Alongside them, Nintendo and Atari gave you 80% of the fun at 20% of the cost. History will view games arcades as a flash in the pan compared to gaming at home.
Namco, Capcom, and Konami went into selling software instead of hardware but it feels hard to imagine buying a home copy of Fable** for $X,000 any time soon.
āø»
* Thatās for video games. In fairground arcades youād literally pay for tokens then use tokens for play. The analogy almost writes itself.
** The model, not the video game.
today at 6:49 AM
If you think the idea of buying Fable like model is hard, you should really try some of the latest open models like Qwen 3.8. Theyāre getting there very fast.
today at 7:02 AM
Qwen 3.8 and GLM 5.3 are my workhorses right now. Qwen for simpler stuff because it is wicked fast using 'ninfer', GLM 5.3 because it is very good a more complex stuff and less likely to get stuck so I can let it run for much longer without intervention.
today at 8:41 AM
How do you run GLM 5.3?
today at 3:33 AM
How much did you shell out for this setup? Whatās the TTFT and the output tokens/sec rate? How much electricity does it consume?
today at 3:39 AM
Working on a blog post to document it all, but that will take a few more days. The short version: the big rig is probably $25K, ttft is dependent on how much prefill you have but typically a couple of seconds (2K prefill/sec in one setup, about 500 in the other), tgs 100 to 150 or so depending on what it is doing and how well the speculative decoder works with the current context. Electricity is pretty bad, full tilt it's about 2500W or so but I've power limited it to 1500 to keep it civil. It is not cheaper than using a frontier lab (or at least, I don't think it is), but the independence bit is the important part for me and I don't care if that carries a surcharge, even if it is a substantial one.
today at 4:13 AM
I'm not saying your approach is wrong - but for my workloads, it makes a lot more sense to just pay for a decade (!!!) of Claude Code 20x Max.
today at 5:04 AM
My local model never says no. I never get banned. If I want to do security stuff, reverse engine, there it goes! Do I wanna chat about politics? No problem! Do I want it to analyze medical stuff? I don't get to hear that it's not a doctor and I should call 911. Do I want to explore legal docs, no problem! There are lots of things that cloud models will say no to! I read that don't even agree to help with advanced biology and chemistry topic much. I also never run out of tokens, no 5 hr limit. I can run it 24/7. I don't run out of tokens. I can run 16 sub agents at the same time, non stop! 1-4 billion tokens a month? No problem. Any agent harness I want! No restrictions. Privacy all day. My own true system prompt, not something someone has determined, no filters. Furthermore, there are many open weight cloud models that don't have equivalent online. For a while before cloud models got good at math, the best model at math was DeepSeekv2 Math and there was no API. The only way to use it was to run local. All sorts of audio models, image/video models, etc. No multiple subscription needed. Private UI for my family, kids included (I can guard what kids infer). Got a friend that really needs some tokens? I give em access through private VPN. Telling someone that they are paying for decades of Claude for local AI is like telling someone they are paying for decades of public transport when they buy a car. Sure, you are right, but ...
today at 7:40 AM
Also no political censorship, and it doesnāt report you to the police when it suspects you did something illegal. That really happened, by the way.
Now I want a local LLM server :( $25k is just so much money.
today at 8:31 AM
MacBooks with large RAM and optimized for local models should be Apple's medium term goal.
today at 8:28 AM
> Privacy all day
For me it's that, and the peace of mind that comes with it. Especially when health related.
today at 7:26 AM
> My local model never says no. I never get banned.
So much this. Plus everything else you mentioned. The freedom and control is real.
today at 8:42 AM
I think both of yāall glossed over the $25k of it all pretty quickly. This isnāt remotely an option for anyone.
today at 5:06 AM
Exactly, that's the spirit. I pool the resources with some friends as well.
today at 8:39 AM
Do you take into account the token allowance? For a heavy user this would be the equivalent of many subscriptions and they add up.
today at 8:30 AM
Keep in mind that the prices of the big AI companies are currently mostly loss makers. Their plan is to get sufficient market saturation and dependency of their clients and if they achieve that, they will hike up the prices quite a bit.
(Of course this all depends on if their plans work out, but if not then the smaller local models will probably play a big role in the failure of their plan)
today at 4:38 AM
If I want to go out of business that is what I would do. Most of the data I have here could never be shipped to a service provider, I'd be in lawsuits for the next decade or so and die bankrupt.
today at 4:48 AM
Okay man I've got to ask, what kind of data are we talking about? Legal? Medical?
I've heard that there are legal software platforms like Clio that offer AI products, not sure what just eavesdropping on some lawyers at the bar.
today at 4:51 AM
http://www.modularcompany.com/
Companies pretty much give us the full overview of what they have from a technical perspective (and often much more than that), we sign NDAs with investors that have serious teeth in them. The data they share with us would not be disclosed under any other circumstance and even our reports are shared only with the clients.
It's a nice little business. I'm not going to risk offending my customers for convenience or to save some money.
today at 4:56 AM
today at 5:29 AM
Holy fuck you have a lot of karma
today at 6:34 AM
He's been here for a loooonnnggg time. I'm pretty sure I found HN through one of his blog posts.
today at 4:53 AM
today at 8:05 AM
I was thinking the economics of it the same way.. if the AI companies are burning debt and VC money to give me cheaper tokens why not take them for it? I can always rent a GPU server for those personal, private AI needs.. heck even run a decision model to route them to the rented GPU servers either on demand or scheduled for few hours in a dayā¦
It doesnāt have to be black or whiteā¦
For my toy projects, adventures, learning exercises, and throwaway MVPs I will prefer the subscription plansā¦ they are cheap for what they offer today.
today at 8:18 AM
From this article alone it should be obvious that the long-repeated myth about frontier inference being subsidized is just that: a myth.
It's training and tuning new models that cost them money. And that's why they are trying to scare people into thinking AI will kill us all in the next 5 years; they want bans on the open-weight models because they know they won't be able to compete with them.
today at 8:12 AM
Yeah, the way I see it short term is take the free/subsidized tokens while we can and the further out we go the cheaper compute will get for local setups.
today at 4:32 AM
LLM subscriptions are heavily subsidized. Iād be surprised if Claude 20x costed the same after a decade.
today at 7:25 AM
Inference prices are going only down. It could be that some top subscription costs more in a decade, but I fully believe a full array of competent models will be extremely cheap.
today at 7:40 AM
The inference is profitable. Itās the training which is really expensive and not being reflected in the subscription prices.
today at 7:27 AM
Doesn't deepseek make $38 per GPU-hour? As in they can actually serve profitably?
today at 8:16 AM
Deepseek doesn't have subscriptions, you can only pay per token / API prices.
today at 6:41 AM
If you compare to API prices the value difference is not as big.
The anthropic sub is likely still subventioned with VC money. So this may not even be available in 2 years.
But yeah, with current sub prices/budget... It's hard to go with anyone else unless you're rich.
today at 6:38 AM
If you compare to API prices the value difference is not as big.
The anthropic sub is likely still subventioned with VC money. So this may not even be available in 2 years.
But yeah, with current sub prices/budget... It'd stick with it too
today at 4:42 AM
The bet we (self-hosters) are making is that once these companies IPO, they will be forced to do whatever they have to do to make money. i.e. price increases, more throttling of subscription non-usage-based accoutns, etc.
They are subsidizing all of us to the tune of $billions in losses per month. That can't go on for a decade. Probably not even for another 18 months.
I still use all the tokens Claude will give me on the max plan. But I also run deepseek v4 locally for overflow and for... contingencies.
today at 4:49 AM
If you know you will pay Claude a few thousand bucks a year a decade adds up too, owning the hardware could just as well end up cheaper because you can sell it or as others have alluded the price to rent could get much worse.
today at 5:10 AM
Would you really expect to be using the same hardware over the course of that decade? One of the biggest drawbacks to using on-prem hardware is keeping up with the upgrades. Leasing upgrade cycles are a thing for businesses, but for someone at home it seems like later in the decade the local system will be showing its age.
today at 5:38 AM
The 3090's I have would sell for more than new today... for the next couple of years or so it will be pretty bad on the hardware front. Until there is more RAM manufacturing capacity and NV has a credible competitor I don't think it will get better.
today at 4:51 AM
Heh, the hardware I bought I could easily sell for 2x so you may actually end up making money. Though I hope that that won't last.
today at 4:49 AM
The benefit of the agent working firstly for you rather than for Anthropic or OpenAI will become increasingly apparent. We're already seeing signs: remember when Anthropic openly promised to have Claude sabotage people using it to create competitors? And OpenAI is looking for revenue opportunities from advertising, which can't end well for their users.
Just think of how user-hostile other online services have gotten, and imagine how acutely painful the proprietary LLM providers could make theirs. Thankfully, open models provide some counterbalance.
today at 4:09 AM
> It is not cheaper than using a frontier lab
Are you comparing to subscription prices or API prices? Because if it's the former it won't last forever.
today at 4:38 AM
True, they're subsidized, I'm not. But I don't have their utilization so I figure it would always be more expensive to run this locally.
today at 4:20 AM
I mean, I just have 4xV100S GPU and a Macbook Pro w/48GB of RAM and I find it kind of amazing.
My server can run Qwen Flash Next at 500 t/s aggregate (about 200 t/s single user). It's pretty great. I use it for local coding and all my worker agents. It is about as reliable as Sonnet 5 for agentic coding. I make 3D models, keep the home network and server updated, fully built ansible files.
You do NOT need 20K in hardware to run these models reliably. I built my own inference engine from scratch for fun. You can run Strata on 16-32GB cards at very respectable token rates if you have the standard RAM to go with it. I strongly encourage people to go use this model on OpenRouter and try it out for tasks.
I also have a somewhat lobotimized quant of GLM Flash 5.3 working and it's pretty decent too.
today at 4:40 AM
V100S is a very good card for some workloads. And they were cheap as well.
I have 240G VRAM total which is a good reason why that rig cost as much as it does, and if I were to build it at today's prices it would be even more expensive.
today at 4:44 AM
I got mine just a few weeks ago for about $750 each. I am a decent programmer and have Opus optimizing the inference engine. I know enough to keep it on track, but GPU programming is something I have literally never done, before, but now I know enough about Volta and how to optimize it to be dangerous.
Kind of ironic, but it's the only way. I just really like this as an option. For a household LLM I have the IQ4_K_XL quant and plenty of capacity for everything you could want in a household LLM and it is trustworthy enough to not rm -rf /, etc.
And yeah that is a really nice amount! You can run and compose a lot of interesting models, vision capabilities. IDK. Plus this level of model is far more interactive in the sense that it can implement a well scoped ticket or unit of work but its not going to magically just figure some things out without guidance or a spec :)
today at 4:59 AM
You got lucky :)
I tried getting some 170's when I first heard about that unlock but they were already unobtanium by then. That would have been a nice way to add a lot of VRAM.
GLM 5.3 runs pretty comfortably on this rig, even with large context.
today at 4:52 AM
[dead]
today at 6:19 AM
At work two different departments have it on their radar to do - apparently there are GPUs waiting to be used for something like it.
today at 6:46 AM
Smart move to get them bought ahead of time.
today at 6:48 AM
8 gpus is cheap now?
today at 7:06 AM
Well... that's a tricky one. I bought a lot of these second hand. I can sell them for more than they originally cost new, so indeed, it is not 'cheap'.
Then again, I run a business and for the business it is absolutely manageable, it is roughly the price of a single job. Privately this would be a different matter but I assume that if you would do this privately you'd simply get a single (but powerful) GPU second hand and offload to RAM. Not as fast but with llama.cpp, 'ninfer', DwarfStar and various other clones of those two you can get some pretty decent performance out of single GPUs.
I've been buying powerful hardware since the 90's, starting with SGI and then ever larger Linux boxes. I make my living with them so it's never been a huge decision to invest some of the proceeds back into the tools. But on an absolute scale, yes, it is a lot of money.
today at 6:11 AM
Depends on context of what people build.
I mean IBM already has a track record of building gigantic amounts of garbage and over charging large corps and govts for it coasting on their brand name and networks to get deals. So using some cheap ass model fits their MO. Having used their shit mountain called websphere, what an IBM CEO says is not something I take too seriously.
today at 7:10 AM
IBM has two faces. Their consultancy arm is the exact opposite of what I would call a good company, even so, they make a good bit of money. The hardware side of the business is still quite impressive, yes, it is too expensive but their power rack boxes are effectively a little mainframe (check the IO stats on those things) and the performance is - I have to admit - impressive. That said I would never buy it because I can get much better value elsewhere. Even so I think they could do this if they really wanted to.
today at 8:26 AM
They do have big advantanges because many large corps have become totally dependent on them having outsourced everything and lost internal knowledge. That internal ops knowledge is very crucial/useful to take full advatange of what AI offers. In the coming years much of it will leak into frontier models. After which corps might not need IBM/TCS etc and just rebuild internal teams to handle ops like they once did.
today at 3:42 AM
> Sure, they use a few more tokens and the results may be slightly less good (but this doesn't really matter because I'm still in the loop so it is easy to correct)
I can't wait to use models that are 'slightly less good' when I am at work instead of something like Fable that lets me focus on the problem at hand as much as possible rather than also spending additional effort correcting for the tool I am using...
today at 3:55 AM
You need to reread what was written. No one is proposing to take away your right to leak proprietary edge to the rapacious frontier labs that need to eat most of the economy in order to survive.
today at 8:36 AM
I think, despite the desperate attempt at IBM to remain relevant (and in business), their CEO has a point here that do you really need SOTA models for everything?
I had a really nice time with GLM5.2 for a month or two - it was, in my experience, approx Opus 4.5ish level of capability when I think Opus 4.8 was the new hotness
Even today, I use Gemini 3.8 Flash as my daily driver for most coding. Its fast and good, even if it is not the absolute SOTA at coding benchmarks it really is very good. Yes I have tried Opus 5.5 and 4 Argon etc and yes the are better, but do I need that?
I think that we are going to be at the stage soon-enough where models are "good enough" for 99% of people's needs. The SOTA will keep on marching ever-onwards I am sure, but I would not be surprised if people stick with previous-gen/previous-previous-gen/etc models because why upgrade?
If this does play out like this, I do wonder how that works out for the frontier labs (specifically OpenAI and Anthropic, less so Meta & Google) - if open models or even just cheaper Sonnet-level models from a few generations ago are good enough for basically everyone's daily work, where is the revenue? how can you justify pumping billions and billions into new generation models and datacenters etc when there is no driving need to use them?
today at 8:40 AM
What subscription do you use for Gemini 3.8 flash and how much usage do you get?
today at 5:24 AM
You can try open models via OpenRouter and omp without having to buy or rent any GPU. I routinely try open models to see where they are. GLM 5.3 with DSFV4.1 as advisor via omp feels Opus 4.6 level, i.e. totally able to ship stuff.
I do use Anthropic and OpenAI mostly because it costs me 1/40 and 1/50 of the api prices, which is way cheaper than Chinese subscriptions, but Iām ready to jump ship when they pull the plug. Fortunately they have nothing that forces me to keep paying them, and I donāt pay the crazy enterprise subscription costs, just the personal account prices.
I also die to be able to run the OSS setup locally to stop being spied on all the time.
For context, I averaged 10 hours a day prompting LLMs from when ChatGPT came out. At the start it was just function generation, then single file modules, then modules, and now I reason, design, implement and test with them.
An interesting thing is that with omp you can plan with a smart model and then implement with a dumber one. So for example, plan with Kimi 3.1 via OpenRouter and then implement with DSFV4 with zero data retention active, sorted by cheapest, and minimal tps accepted.
I get the people who are pissed about their cushy jobs evaporating, but if you know how to use these technologies itās very freeing because itās like having a company of people working for you, except they are not human so theyāre not wasting their life working for you instead of using it for themselves.
You can make them do pretty much everything that is fuzzy-verifiable, you donāt have to worry about their feelings or that they wonāt do what you asked them to do or stop, and you can activate them when you want at scale in parallel or not use them at all for days.
For me itās a dream honestly. I never felt so much empowered in my life while being enabled to unlock my creativity with almost no boundaries.
today at 8:06 AM
I fall into the people pissed about losing jobs camp, can you explain how you keep a cheery attitude like you do?
I geniuinely don't see what I could make with this new tech, my brain immediately goes to "this app idea is worthless, anybody can make it now".
today at 8:21 AM
As the OP said really. It is very very freeing to just be able to give a task that in ye olden days you knew was going to be a pain in the arse to do, and it will do it without you. Amazing.
It's like having extra copies of yourself that you can just fork and get stuff done. E.g. the other day I was in a meeting and there were some heavy discussions about a few gnarly issues. I fired off a few prompts in parallel while the AI researches options, approaches, made a prototype etc all while I was present and paying attention still in the face to face meeting. I am so much so much more efficient as an engineer now - getting better feedback, more bonuses etc. It's never been better - I handle thinking about the requirements, the computer science and software engineering big picture etc, and the AI handles the syntax trivia for implementation.
The models are really really good now.
The only downside is that as a result of more stuff getting done, we need to have more meetings to discuss what actually needs to be worked on.
today at 8:20 AM
Think bigger. Don't think in features. Think in business workflows.
You are a consumer, so you naturally think like the consumer you are. But consumers want cheap stuff and only big big businesses are able to provide cheap stuff at massive scale. So stop thinking in b2c. For us little businesses the money is in b2b. Businesses pay lots of money to have problems solved. They want to focus on their core business and outsource the rest.
Consumers hate paying subscriptions. Businesses love paying subscriptions. Because then they can make their problems someone else's problem and focus on what their actual business is.
today at 8:13 AM
Making an app has always been approximately 1% of the work involved in turning an idea into a business.
today at 6:23 AM
In OMP just setting the "task" role (which controls the default subagent model choice) to some cheap Flash model like Deepseek or Gemini goes a long way towards making better use of your quota.
Sol and Opus both are quite eager to use subagents and since the subagent tasks are often well specified and then verified by the main model there's hardly any advantage to using anything better for the subagent anyway, so IMO it's a no-brainer.
For me switching the main model to Kimi K3.1 Pro would put me towards 100% open, but I don't know if it uses subagents effectively. Any other open models I noticed just don't utilize them very much.
today at 3:03 AM
But hardware to run them is conveniently something IBM would be happy to sell you.
today at 3:30 AM
If IBM would ship a piece of affordable hardware that can run large LLMs efficiently I'd be lining up.
today at 3:33 AM
"Affordable" and "IBM" don't belong in the same sentence together, but they're happy to sell you their inference-capable hardware if you've got the wallet for it.
today at 3:37 AM
I don't agree with that. They made the PC revolution possible by making hardware that was at least affordable enough to be bought by a large number of businesses and individuals. It also set the stage for the clone market. I'm pretty sure that if IBM really set their mind to it that they could do this.
That doesn't detract from your overall point that they tend to make overpriced stuff, but I don't think they have to do that. Besides, what's overpriced? NVidia's GB300 workstation sells for 95K euros, a single RTX 6000 is now 20K. Surely even IBM can compete at that level.
today at 3:45 AM
That was completely accidental and they themselves tried to close the barn door with the PS/2 line in 1987 but at that point it was too late.
today at 3:56 AM
True, even so: They still make some pretty impressive hardware. They have access to fab capacity, they know how to make chips. They have the capital. If they really wanted they could do it.
today at 4:25 AM
In what way is the IBM Spyre not what you're asking for?
today at 4:30 AM
In a way it is, but I would see that as a starting point, not as the ready solution. They'd need to do what NV has done for their eco-system: fantastic libraries, a credible future upgrade path, easy ordering, manufacturing partnerships, the works. I figure besides the performance gap that they are at least 5 to 10 years behind NV with their present offering. But NVidia is dropping stitches and IBM should take advantage of that.
I think IBM sees the Spyre as a reason to sell more Power hardware but that's just not where the industry is right now and I doubt it will be enough to create sufficient demand. Who knows, I don't have any insight in what they're doing there, but there is massive gap between A GB300 workstation and what IBM currently offers. At the same time: the RAM crunch hits all parties and that's a major obstacle as well.
today at 4:46 AM
Yeah. For all the jokes we make about paperclip maximizer, we've figured it out. It's not paperclips, it's RAM.
That's the thing though. Software is cheap with AI, but Nvidia has Cuda. If software doesn't matter anymore (it does) because it's so cheap now, then Nvidia's moat evaporates to IBM's Spyre. In reality "just a little bit of software" has inertia and cannot be dislodged quickly, quietly, or easily. cf Microsoft.
today at 4:49 AM
That. And not starting every sale with 'call us' where they first want you to tell them how much you can afford to spend. I buy a fair amount of gear and if the sales process starts with some sleazeball sales guy instead of a mouseclick on a local webshop then I'm out.
As for CUDA: I think CUDA is a formidable moat but I also think that that is precisely the kind of thing where the likes of IBM could (if they really wanted to) make things happen. They're already in the space with Redhat and they have ample knowledge about drivers and low level code. Putting together a team that either makes a drop-in replacement or alternatively, that starts supporting their custom hardware for the big players in the inference space (VLLM, llama.cpp and Torch) would make all the difference. They'd have to keep doing it though, even if the initial results were lower than expected because that market is not going to turn on a dime.
today at 4:51 AM
Respectfully, those people don't work at IBM anymore. The organizational memory doesn't last that long.
today at 4:55 AM
You'd think that with a 250K workforce and an active line of hardware including an AI accelerator that those people are still working there. I could be wrong, but someone has to be doing that work unless IBM in turn outsources it.
today at 3:42 AM
They havenāt been _that_ IBM for decades
today at 3:50 AM
I know. But: they have gobs of money, they still have some talent, they still have a hardware department that I'm pretty sure would jump at a chance to give NV a run for their money. This would require a massive long term investment and a commitment that I'm not sure they would have the stomach for.
today at 4:31 AM
"they have gobs of money, "
They do? They are a spec in the ocean relative to big players offering inference and training their own models. Pretty sure one thing they are struggling with is the money to compete successfully in the ai game this time around.
today at 4:35 AM
10 billion give or take, market cap 250 billion, 250K employees. That's not a 'spec in the ocean' by normal standards, and IBM is a mature company, not an inflated balloon.
today at 3:44 AM
> They made the PC revolution possible
Yeah they were only half a decade late to the PC revolution back in the day...
today at 3:58 AM
They weren't really late. In many people's minds IBM validated the concept of the PC.
today at 4:02 AM
When that happened I'd been working at Tandy (EU version of Radio Shack) for a while and even though we sold computers they were not taken serious until IBM came along and then suddenly we couldn't supply them fast enough.
today at 4:22 AM
Was Tandy only EU? In the TRS-80 days the name came up a lot, although maybe just because of the acronym ... 6809E was a good time
today at 4:25 AM
Yes, that was the first time I had a 'proper' processor to play with. A Dragon 32 (CoCo clone from the UK).
I've never seen Tandy used outside of Europe but then again, I had not traveled as much then as now so it's definitely possible they used the brand on other continents as well. I wonder how many trees were cut for those catalogs :)
today at 6:50 AM
And AU. Dick Smith started out selling Tandy's, before switching to their own clones.
today at 4:28 AM
It's the "PC-compatible" hardware make those affordable, not PC.
today at 5:08 AM
I didn't think that IBM had gear that mortals would want to run. If you're not running a mainframe, what do they have to sell you? They still sell Power servers and it looks like they have an AI card that can be used on both. But, if we're talking about the IBM CEO talking about on-premises, what are they really talking about hardware-wise? It's not something they would sell, is it?
This is more of a statement for their consulting customers, right?
today at 3:44 AM
Yes but unironically? IBM should be what NVIDIA would be competing with their NVL72 and DGX products. It doesn't make sense that IBM is just watching NVIDIA take its lunch.
today at 4:45 AM
I think they will need a far more aggressive management before such a thing could happen. But I can dream about it.
today at 4:27 AM
> It doesn't make sense that IBM is just watching NVIDIA take its lunch.
If you have ever worked at IBM, it makes total sense. Completely dysfunctional middle management and on up.
today at 3:11 AM
Not till Jensen buys his next leather jacket.
today at 5:41 AM
Of course IBM - the guys that sell you mainframes - would be saying this.
This is a win though. But not for the reasons people might think it is.
It is a win because it's a massive player now countering the new (temporary) behemoths. With all the leverage, budget and connections.
And contrary to the AI Labs, IBM isn't just that one SV rockstar dude, but a whole org with tons of people. Meaning that they're going to be far closer to the baseline of sane humans just from averaging alone.
today at 5:46 AM
The guy that sells mainframes also gives away the Granite models for free. I've found them to be quite capable in content retrieval and long-running workflows, even (or especially?) the small ones.
today at 3:40 AM
Considering the disclosed numbers for DeepInfer and the usefulness/popularity of DeepSeek 4.1 Flash, I expect it is inevitable that more companies are going to consider the self-hosting option seriously, and perhaps play with the idea of offering inference as a vendor.
today at 5:47 AM
[flagged]
today at 3:21 AM
While he thinks most businesses can build useful AI infrastructure in just two or three racks, he feels many will struggle to get it running.
āThe question becomes: how do you do that?ā he said. āThe question that the enterprise struggles with is: āWhere are my kernel engineers? Where are the people who worry about the underlying network? Who can make sure that the model is clustered on GPUs that are close to each other in network proximity? Who can make sure that youāre doing the right caching of results so you get an incredible response?āā
today at 4:35 AM
It's not a one time thing though - onprem inference requires continuous monitoring and tuning to get the most out of your GPUs. Looking at 25-50% performance benefits. Today, it does require dedicated Inference Engineers...
today at 3:34 AM
That's based on the idea that humans can learn and adapt and solve problems, and that's not what companies want to think about when they're seeking a profitable transaction such as hiring.
today at 3:35 AM
Do not do that if you can't prompt it correctly. It is a potential disaster waiting to happen. LLMs are likely to pull things off the shelf without real defense in depth and that's a problem. Maybe slightly less if it's only in an internal network, but still.
today at 3:44 AM
RIP frontier AI, the top customers can easily self host
today at 4:03 AM
What happens when their competitors decide to pay for a more intelligent frontier model?
today at 8:30 AM
The target audience is likely those who can't or won't use frontier models unless they are on-premise.
today at 4:15 AM
What happens when your competitors invest in better and more expensive tools or materials in general? They (potentially) make better products, but those products also cost more. It's turtles all the way down.
today at 7:49 AM
Cool! That's when you pay the difference in marketing and auto-win your sector! People don't actually buy the best, they buy what they think is.
today at 5:43 AM
You buy the competitor and kill their culture from the inside, ruin them and take a tax loss on the company.
today at 4:30 AM
depends on the economics and much more, the best Ai cannot do a lot of things, 90% of human tasks by some reports
we technologists tend to extend what we see and think to other realms where it doesn't make sense
today at 3:17 AM
Can we please stop giving weight to CEOs talking their book? Like of course IBM wants you to think this, it would be great for them if itās true. And maybe it is true! But IBMās CEO saying itās true tells me just as much as if a frontier lab said itās not: nothing.
today at 3:23 AM
"CEO Said A Thing!" Journalism
https://karlbode.com/ceo-said-a-thing-journalism/
today at 6:06 AM
Great term. Plus the related "Politician Said a Thing!" journalism
today at 4:28 AM
thank you, I was trying to re-find this the other day
today at 3:27 AM
But you know - it would be funny if somehow Open Weights are the thing that completely save IBM
today at 3:34 AM
Linux already served as a life-preserver for them also.
today at 7:48 AM
It was a brilliant move at the time for IBM, and has helped them shift their business more into providing services and support.
I imagine they are panicking a little with LLMs reducing the reliance on external companies for software support, so they are really hoping to shift it to on-prem AI, which is where the bias in his statement comes from.
today at 3:23 AM
Itās a conversation point off which a discussion can be had
today at 3:32 AM
the only credible "conversation" is "well he would say that, wouldn't he?"
today at 3:32 AM
"we are delighted..." said a spokesman
today at 4:07 AM
Iāve been in tech for a long timeā¦ I have no idea what IBM does. Now, I get that Iām not the customer, that their customer is ābusinessā but not my businessā¦ noā¦ bigger business. The biggest of businesses I assume. I think it must either be a club Iām not in, or IBM is three half racks in a trenchcoat submitting invoices for millions to companies where people just pay it because āitās IBM, it must be importantā.
today at 7:50 AM
Review the specifications for the latest Telum II chip if you need a hint. 5.5ghz base clock.
If you absolutely need to scale something vertically, IBM Z is the final stop on the line.
X86 is great at hosting an entire farm of applications. You would never want to install salesforce or gitlab on a mainframe. That would completely defeat its purpose. If you've never encountered an appropriate workload, IBM might never make intuitive sense.
Also, the roadmap:
https://newsroom.ibm.com/2026-08-24-ibm-unveils-next-generat...
today at 4:39 AM
Ironically one of the smaller places I've worked at tried to use an IBM product once - I'm still not clear on what it was meant to do or how it was better than any cheaper alternative, so I assume someone in leadership was fooled by a very impressive pitch.
Lots of work went into integration and it never went live.
today at 7:04 AM
> I have no idea what IBM does
For one thing, amongst many, IBM mainframes process 70% of global transactions on a value basis
today at 7:16 AM
[flagged]
today at 7:33 AM
[flagged]
today at 4:25 AM
[flagged]
today at 2:08 AM
[flagged]
today at 3:13 AM
Good enough for the old world, but I'm not so sure it's good enough for what's coming.
today at 3:14 AM
What do you think is coming that isn't already in the old world?
Cerebras with Claude 8?
today at 3:36 AM
OpenAI most likely pushed Cerebras into a contractual agreement not to do business with Anthropic through 2030. ("Most likely" because the names of the companies Cerebras is contractually obligated not to sell to are redacted in public filings)
today at 4:11 AM
Imagine a regulatory framework so weak that a company feels comfortable to do something this anti-competitive