today at 6:59 PM
This is the golden age of model training. Some days ago, I decided I wanted a local CPU only model that can perform exceptionally well for English to Bash translation (to avoid the googling for command syntax). I got a bunch of subagents to generate large amount of training data (140k+ samples), got the Qwen 3 0.6B base model, pointed Astra at it, and off to the races. It trained for 2 days (on and off) and I got a surprisingly good model for my task! The total active time I spent was a few hours. And it is still improving, what a time to be alive!
today at 7:32 PM
Curious about how you generated the training data? Was it just asking an existing model to generate a bunch of examples?
I ask cause would this be a kind of model distillation?
I have a small model I'm looking to train on some data, and I have some real live data but I'd love to be able to extend it.
today at 7:43 PM
All synthetic data. For this usecase, it was easier because all current generation LLMs, even the small models, are really good at bash commands (and SQL queries too)), so you can reasonably start batches of cheap subagents whose output is reviewed by a more capable model and merge into main training set. After 100k, I had to standing instructions to run the generation loops selectively, meaning only update samples in a given area where we see poor capability.
today at 9:43 PM
Do you have a write-up or git repo for this? Would love to learn more and/or dig into the guts
edit: others have asked any you have replied "soon (tm)", looking forward for that day
today at 8:04 PM
It is a form of distillation, as long as you're working a very narrow "trivial" topics it works perfectly.
today at 8:02 PM
I don't understand. If you have a model that can do bash examples already (your subagents), then why would you need to train a model?
Or are the subagents generating your training data using a closed/paid model?
today at 8:08 PM
A very small, highly specialized model can use negligible resources (CPU, energy) to accomplish the same task.
For everyday work that happens frequently it's better to have a tiny specialized model instead of making billable API calls or turning your laptop into an 80W space heater for 20 seconds to run a general purpose model.
The large models can be used to generate synthetic training data. Tell them to make up 100,000 tasks paired with the resulting output as a 1-time cost. Then use that to train a small model.
Think of it as distillation, but focused on a specific task.
today at 8:26 PM
Given that they're just using it to avoid the googling for bash command syntax, I'm not sure they'll save in the end against the 140k training examples they generated.
today at 9:41 PM
Good observation! It would have to be offset with O(140k) queries to the model, which is, well, unlikely.
today at 10:06 PM
Just like with OSS in general, being able to distribute it is what makes the effort worthwhile.
This particular example is maybe a niche, but 1400 people can use a few hundred queries in a reasonable amount of time.
today at 9:50 PM
If it's about the latency / flow disruption, spending a few hours once could easily be worth it if the result is actually good enough to skip googling/retries.
today at 9:38 PM
you can probably generate quite a few example pairs in a single shot, you also likely don't need the best models for this either
today at 8:46 PM
This is so cool - I'm aware of this in a vague way. Can you write a little tutorial or give some good links. I want this to be the next new things I do :)
today at 9:44 PM
Better yet package it up in a skill!
today at 8:08 PM
The models he is using to generate training data are presumably commercial models. He is distilling their bash knowledge into a much smaller model he can run locally fast and cheap.
today at 9:50 PM
Did your Astra do any RL or just SFT? did it make up any benchmark to ensure the fine-tuning was a success?
today at 8:17 PM
If itās one of thing that you want just for English to bash shell commands, I will create AST, it is deterministic, exceptionally fast, no tokens so no need to fine tune existing model, please let me know your thoughts.
today at 7:45 PM
That's a really impressive result. There are all kinds of small tasks like this I use an LLM for, but theoretically if you broke all the sub-use cases into local-only models, and had something lightweight that routed to the right model, you could have faster and cheaper workflows. E.g. something trained on the linux man pages for common commands, since it's usually quicker to ask an LLM for a specific command with flags than to consult the man pages.
today at 8:24 PM
> That's a really impressive result.
we dont know what the result is and how its impressive.
today at 8:37 PM
Sorry for the aside, but I noticed half the usecase of AI is fixing the awful DX.
today at 9:06 PM
I appreciate the aside. Interesting observation
today at 9:41 PM
I'm literally working on context/harness engineering right now (a set of opencode plugins)
Aside on the aside, I welcome this new era of really personal software. Not Ai's being sycophants, rather being able to easily and quickly change, adapt, or extend software I am not familiar with.
today at 7:06 PM
what hardware are you using to train?
today at 7:10 PM
I didn't have a local GPU, so I asked it to go out and find hardware. It found a google TPU v6e which seemed reasonably priced. I gave it my google api key. I told it to use TPU only when training and bring it down afterwards. That's about it.
today at 7:40 PM
> I gave it my google api key
This is the part where the narrator looks at the camera and says "Don't try this at home, kids!"
today at 7:47 PM
Youāre absolutely right, I shouldnāt have rented a 200 GPU cluster for $35,000/hour. Thatās on me.
[Search: Can I refund Google cloud?]
It looks like weāre not able to ask for a refund since we did actually use all of that compute intentionally.
Would you like me to write you a pleading email to send to the support team?
today at 7:47 PM
Thereās a safer way to do this with nearly no added friction. Give it a read only API key. Then just ask it to write the API calls into a bash script and then read it and run it yourself. The agent can still inspect the live resources and diagnose and give you more commands to run. I do agree I wouldnāt give it create / write access.
today at 7:46 PM
Why? Isnt the API key scoped to a project and specifically made for this?
Are you confusing this with an OAuth token or something?
today at 8:02 PM
Until astra goes bonkers and use the tpu for days
today at 9:44 PM
this is what billing caps are for
https://docs.cloud.google.com/billing/docs/how-to/budgets-sp...
today at 7:53 PM
I've done this sort of thing before but with Vast. Pre-deposited some money online, then let the LLM request and manage a training run on an allocation. Worked pretty well without risking bankruptcy.
today at 7:18 PM
What kind of observability did you have over this process? Iām interested in how my peers are operating these efforts.
today at 7:29 PM
On the cloud side, nothing valuable existed, so the training couldn't ruin anything it didn't create. On the laptop side, I usually ask the agents to create named scripts for everything it needs to access, then those local script directory is green-lit with approve all. For cost, I kept giving it new budget in the 20-30 dollar increments.
I had to intervene a few times. For instance, as smart as the models are said to be (Astra), it would copy the full training run, train on the server, pull every checkpoint to the local machine, then run tests, update. So, the bandwidth bill was as high as training bill for the first 6 hours. It could have simply tested each checkpoint on the server, saved time and money, didn't occur to it until I said.
today at 8:40 PM
Perhaps I wasnāt clear. What kind of instrumentation and alerting, if any, did you employ to keep an eye on it?
today at 7:24 PM
today at 7:34 PM
> I told it to use TPU only when training and bring it down afterwards.
I wouldn't put my house on it. Brave.
today at 7:15 PM
Neat!
today at 8:36 PM
> what a time to be alive!
It's good to hear you're enjoying yourself, but I suggest retiring that expression. It's really beginning to grate.
today at 9:29 PM
Seriously, I'm using a Qwen 3.8 27B on the homelab, distilled from supposed Fable traces. Regardless, the difference is notable, less thinking, better output. Distilled / heavy quant is better than the original (imv)
https://huggingface.co/vwdubb/Qwen3.8-27B-Fable-Distill-NVFP...
side quest, are fable distillations only wrong when it's another country?
today at 7:08 PM
i also need more info!
today at 7:11 PM
I am thinking about opensourcing everything, although this is not my main domain or my main startup, so the overhead of huggingface etc seems a bit unnecessary
Edit: will do as soon as possible
today at 7:27 PM
just ask the agent to write it up if you don't have time to do a write-up yourself
today at 7:45 PM
+1, would like to see. Even if it's not fully "ready for consumption", it's probably enough to reproduce the results.
today at 7:54 PM
Please do! Small, specialized models need more love and the time you spent would be a gift!
today at 7:56 PM
Would also love to read a write-up about this!
today at 8:32 PM
Golden age before the age that ends humanity. Not talking about any "rogue AI", just the known statistical models of what is coming due to climate change.
today at 9:39 PM
Do those statistical models account for declining birth rates or are they based on prior population growth projections?
today at 10:15 PM
I don't see why I would be interested in this model, considering the price difference. They advertise that it's the same as Kimi K3 in half the tokens. But the pricing is double the pricing of K3. So why do I care if it uses fewer tokens, if I'm paying double per token?
today at 10:25 PM
Because you care about how many tokens are used per task. What you said is like only caring about the price of gas and not gas mileage of your car.
today at 10:30 PM
it's not exactly the same, the model stills "weighs" the same
here, it does less work, it's more like driving half as far but still paying the same total cost
today at 8:33 PM
Itās the first time I know fireworks has a team doing model research. I do have a complex mood in that. On one hand, Iām always happy to see improvement of OSS models, whether thatās on intelligence or cost-efficiency. On the other hand, I would be a little worried about using fireworks as my API provider. Till the moment I saw this news, I had been using fireworks as my provider of deepseek v4 flash, because I thought fireworks acting as a role deploying OSS models and selling calculation resources, should be safe to use without worry of data being used for training since thereās no āconflict of interestsā. But I would think twice now.
today at 9:20 PM
Just read the terms of service and read this blog post and I think your concern will be addressed.
today at 9:49 PM
this is our preferred open weight token vendor
this work may explain why recent models like qwen-3.8-flash and MiMo-2.6-* have not made it into their offering, which has given me reason to pause my excitement for Fireworks
today at 8:48 PM
That also could explain why openrouter is worth that much
today at 8:43 PM
[flagged]
today at 6:31 PM
Off topic:With sol pricing drop tbh kimi k3ās value prop has not been that great. For our internal use case/testing/benchmarks sol come out with way better quality and much cheaper costs. Kimi really needs to drop their pricing (I heard itās set by them across all the neoclouds) Sol is at 2/10 vs kimiās 3/15
today at 6:45 PM
Agreed. Even on the open weight side, GLM 5.3 has roughly equivalent performance to Kimi K3 for less than half the cost.
today at 7:31 PM
Competition is good. Without K3/GLM/DS4 etc. there would be no pressure on OpenAI to drop Sol's price.
today at 8:25 PM
Sol pricing dropped but so did the quality few days ago. I wonder when these companies are sued for making the terms from their side to go downwards while taking the same subscription cost.
today at 10:18 PM
I am glad I am not the only one to notice. I feel like I've gone back to Sonnet 4 levels of incompetence!
With Sol 6 I am back in a world where the model writes bad code because it is lazy ("You're absolutely right, I did not [do it properly] because I did not want to edit [a normal amount of files]").
today at 8:29 PM
Is anybody tracking these quality changes? All I've seen so far are accusations (quite a few at this point) but not really any actual data.
today at 8:38 PM
I donāt understand how there isnāt a website out there tracking this stuff already.
today at 9:02 PM
how the hell do we even track that? and before someone says...
ā"Benchmarks!"
...I'll tell that they can be gamed so easily, and they are on a consistent basis.
today at 10:11 PM
Sure they are, but do you think they are continuing training to improve a model after release without bumping the version number, presumably only to game the benchmarks?
today at 8:34 PM
In a Codex subreddit there is a bunch of stats.
today at 8:33 PM
today at 9:17 PM
With DeepSeek's pricing, no other value prop has been great.
today at 6:53 PM
Agreed, I think the only place where itās still interesting is ui design. Visually kimi and muse feel much nicer than frontier models to me, but maybe itās an artifact of everything terrible being Claude Design
today at 8:02 PM
I was surprised by that. I run my benchmark [1] every couple of days and was sure this model will be ath the pareto frontier, if not THE pareto frontier. But no:
Ember isn't picked yet. In planning, Opus 5.5 wins under the planning weights. In code, GPT-6 Sol dominates it: also 10/10, but with a higher quality score and a lower estimated cost. Ember has no intelligence index, so its starting score is only 0.73, which holds its 10/10 down to 0.954 against Sol's 0.975.
[1] https://philippdubach.com/posts/jev-model-router-for-pi/
today at 8:05 PM
> With sol pricing drop
6 or 5.6? Because 6 is hot garbage
today at 10:20 PM
Could be a really interesting article but they disabled reader mode so I guess Iāll never know.
today at 6:21 PM
Ignoring for the moment issues of what "counts" as open, won't open models rapidly advance due to stuff like this in ways that it's less possible for the proprietary ones to do? This is exactly how Linux & Wikipedia, for example, overtook their "frontiers", right?
today at 6:32 PM
> Ignoring for the moment issues of what "counts" as open, won't open models rapidly advance due to stuff like this in ways that it's less possible for the proprietary ones to do? This is exactly how Linux & Wikipedia, for example, overtook their "frontiers", right?
I suspect the advantage that catapulted Linux ahead of the establishment was less technical potential and talent and more organizational advantage. That's not to diminish the technical talent of the Linux crew, but them being unencumbered gave them more degrees of freedom. The rest is history.
So as long as the AI companies don't succumb to "big company" dynamics, they can outlead. To wit: Open AI and Anthropic are kicking Google's ass.
today at 7:29 PM
I think people make mistake here, googleās approach is not to spend $2.3 on every $1.0 earned, theyāre riding on serving to masses ālunaā, they absolutely have way more powerful models internally but they donāt clutter their infrastructure with fragile and costly intelligence-of-size inference frontier. I think āunderdogā perception is illusory/temporary, not stupidity - calculated, conscious, longer term bet.
today at 10:31 PM
Where do you get this "not to spend $2.3 on ever $1.0 earned" from?
Google might not have compelling frontier offerings, their chat harness is complete garbage compared to any other lab (in large part due to a bizarrely badly designed harness where something like code execution requires the prompt to undergo some sort of classification step, no idea what they are doing).
But they absolutely kill in terms of usage offerings. Google lets one subscription be used by *SIX* different google accounts on a family plan.
Plus I currently literally get *$40/month* of Gemini API credits on developer.google.com because they gave me a $10/month grant 4 times.
They give you 200 cloud compute units on google collab, this literally lets you spin up an H100 for around 40 hrs or something if you want to try spinning up local models.
You get Jules (huge allotment btw), Image gen, Video gen, Music Gen, antigravity usage, 5 TB of cloud storage, Notebook LLM...
today at 9:11 PM
I tend to agree, but I also should highlight how expensive this shit really is.
in one month, Google actually went cash-negative. [0] even still, they are subsidizing their stuff a lot less, have the most opaque and variable limits, and increase adoption through bundling and shuffling features. I can't even share my Google One storage without subscribing to a Google AI plan anymore, but previously any plan except Google One Lite was shareable.
if you tell me that's not enough to go after frontier, then how much money are Anthropic and OpenAI burning?
[0]: https://www.techspot.com/news/113214-google-records-first-ne...
today at 6:39 PM
Diff people have diff motives to experiment, then new work is done on top of stuff that "hits" in a way no one anticipated. Then work gets piled on top in a way that might make it hard to port
today at 6:40 PM
> then new work is done on top of stuff that "hits" in a way no one anticipated.
Indeed. And when you have freedom to play, you are able to find new stepping stones that you didn't anticipate. And you can combine stepping stones in new ways to make new discoveries.
Greatness cannot be planned.
today at 6:32 PM
No, because close labs/models borrow but don't contribute back.
today at 7:40 PM
This was exactly the crux of Nathan Lambert's recent testimony to a group of US Congressional members/staff: https://www.interconnects.ai/p/the-current-balance-of-power-...
today at 6:57 PM
Won't the "frontier" labs figure out whatever techniques were used and apply them to their closed models?
today at 7:07 PM
Like how the last 2 decades of tech companies are thinly veiled open source pilfering into business units.
today at 9:07 PM
"Oh darn, you know that thing I made and released with explicit, precise language defining who can use it and what, if any, restrictions apply? Well now someone is using it in complete accordance with those conditions I set out, and that's somehow making me upset"
today at 7:04 PM
If they can keep up.
The lock-in is less pronounced as it is with AWS or MS.
today at 6:55 PM
The difference between contributing to OS and AI, is that the first is a hobby alternative to woodworking or hiking, while the other can easily bootstrap you a company you can get millions in investment, at least for time being.
today at 6:25 PM
Yes, absolutely, but only if people keep contributing in the open.
today at 6:32 PM
not necessarily, just knowing something is possible will motivate others to achieve it somehow. Which is why there are so many LLMs and OAI doesn't have a monopoly
today at 8:19 PM
What am I missing here? I think of fireworks as an inference provider serving open weights model. The value that they primarily provide to customers is that (i) they improve reliability by balancing across a bunch of clouds/neoclouds, (ii) they get better pricing by buying capacity in bulk, and (iii) they reduce operational costs. So far so good.
I can also see the argument for providing a post-training service from a customer acquisition perspective: "hey, we can fine-tune this open weights model, so it both gives better/more predictable results than OpenAI/Anthropic and also is cheaper. And btw, once we've won your business, please run this model on our infra."
But what I'm struggling to understand is fireworks spending a bunch of money (on salaries and compute) releasing a frontier model that is going to rapidly fall behind the frontier. Is this "just" advertising for them, both for customers and also for hiring? Or are they actually trying to stay on the frontier? If so, to what end?
today at 8:27 PM
> to what end?
I'd expect that their business strategy is to compete in more markets, and if successful, they can capture more value. This is the "easiest" for them as they already have GPUs, a training environment etc. For that platform it's not the worst if there's an internal customer team that can help shape the future and provide immediate feedback, and if it results in a good model, even better.
Other things I'd not be surprised they offer in the future in the same vein: A multi-model harness, coding agent (cloud and local), and maybe at a later point in time even a CPU-only cloud compute product.
today at 8:25 PM
The end: make lots of money. The means: systematically take existing reasoning models, do some more post-training of some sort to make them achieve the same outputs with less reasoning tokens (ie, cheaper). Same quality but cheaper is always valuable.
It's unclear if they can do this systematically and it's unclear if they can do it better than others. But, lots of things are unclear in AI at the moment, this doesn't seem outrageous on the surface. And, it could just be marketing. And it could be the first option with the backup of the second.
today at 8:21 PM
I think they are trying to show potential customers what is possible.
today at 9:18 PM
Vertical integration.
today at 7:26 PM
Over on /r/LocalLLaMA there's a group that's been getting popular doing the same thing for the Qwen 27B (and other) models. - https://huggingface.co/ukisai
today at 9:06 PM
Given what an experience i had with Ember-2ā¦ Iām not sure Iād want to engage with its predecessor.
https://en.wikipedia.org/wiki/Exapunks?wprov=sfti1
today at 6:43 PM
> The problem: thinking models think too much
This is partly the appeal of Jev et al; having a quick model for simple tasks, that doesnāt require that much thinking
Itās amazing all the workflows that models like that can unlock. And yes, classifiers and other ML models have been around for a while for these types of tasks, but Jev has made it easy and cheap to play and experiment. This in turn, is incentivizing people to try them for a bunch of stuff, unlocking creativity and producing a lot of new cool (and eventually potentially very useful) applications
today at 6:53 PM
What are the useful applications of Jev so far? Not to sound dismissive, I just havenāt seen what people are using it for yet.
today at 8:27 PM
Here's a third-party (not Jev) showcase of things people built, which helped me kind of get the appeal. https://bentossell.com/jev/ (not mine).
today at 6:59 PM
Lots of use cases! I've personally used it for the following:
1. Evals (once you have your rubric defined and tuned using a reasoning model, jev can be great for running periodic evals especially those that run daily.
2. e-commerce catalog classification 3. quick search using anything as context and query mapping to a pre-defined set.
today at 8:13 PM
At least for 1, evils, youād want to use a good old reasoning model to get the best eval results.
today at 7:11 PM
Why not using a cheap LLM with thinking completely disabled ? I don't think it will be much more expensive than jev.
today at 7:24 PM
Iāve tested this with some local LLMs and their accuracy is in general better than Jev/Laya, but they are super slow in comparison as well
For example, a typical/stock LLM canāt really play Doom in real time, but a Jev-like model can. Just because of latency
Of course, if you want the best Doom player, there are way better and faster adhoc models
today at 7:47 PM
LLM inference has two very different regimes of work: prefill & decode. You can think of the former roughly as processing a pre-specified prompt, and the latter as sequential processing (auto-regressive token generation) eg. "chain of thought". The latter is very important for LLMs and cannot be ignored; it deeply influences infra design, even necessitates copious amounts of high-bandwidth memory. Jev-like models can ignore the latter and therefore optimize much better for the former, consequently operating at both better cost and latency.
today at 6:19 PM
> The problem: thinking models think too much
Analysis paralysis stifles not just human intelligence, but other intelligences too.
today at 7:27 PM
The thinking traces on some Chinese models just output the full response in the thinking trace, then output it again to the user, which is redundant.
today at 7:19 PM
Yes and thar makes you wonder if the Paradox of Choice would apply as well ;)
The more options you have, the harder it becomes to be satisfied with the one you picked.
today at 6:24 PM
So they trained a model on open weights, and then aren't releasing the weights... am I reading this right?
today at 6:34 PM
Technically kimi k-3 weights license is not open weight (it has a lot of restrictions). I would classify it as āweight openā similar to the bsl and fsl āsource openā licenses.
today at 6:45 PM
weight available
today at 6:45 PM
It happens. Most open licenses aren't GPL style copyleft.
today at 7:00 PM
There is little to no point reading the article as well. It's stripped of all alpha.
> task and environment feedback
> on-policy planning and learning
> feedback connects decisions to their consequences
These are deliberately the least informative phrases you could possibly use to describe what you have done, while still being in the realm of words that go over a generic investor who has no idea whats going on and may be dazzled by sciencey sounding language.
Cursor compose 2.5 article where they used and described on policy self distilation was actual alpha.
today at 8:50 PM
This is precisely my point.
today at 6:57 PM
Aren't Cursor Composer models like this too? At some point all the extra RL you do can be considered as proprietary information added.
Not suggesting this is right or wrong, but is sort of the nature of the technology.
today at 7:08 PM
today at 6:55 PM
It happens with open source software all the time, why would we expect any different with open source weights.
today at 6:57 PM
Because we do. The GPL isn't a suggestion. If you can take open source code and make private software out of it then what are we all doing? No, license requirements and agreement are law for a reason.
today at 7:13 PM
Kimi K3 has its own license which is permissive, it isn't at all like GPL https://github.com/MoonshotAI/Kimi-K3/blob/main/LICENSE
today at 8:17 PM
GPL is a specific license, itās not FLOSS as a whole
today at 7:19 PM
Because the licenses that apply to software make no sense in the context of LLMs. With the latter, there is no source code to license.
The words of a license are what the license is.
today at 8:16 PM
Which is fine, thatās legal according to the license
today at 6:29 PM
today at 9:12 PM
This is undoubtedly great. But most of the inference cost today for dominant use cases (agentic coding) are in the prefill, not the decode. This is one of the reasons that DeepSeek is so aggressively optimizing prefill and caching.
today at 8:23 PM
Been thinking about the feasibility of training a model using synthetic thinking traces that were reduced to caveman-speak prior to being used for training. Seems like it would be fairly easy to generate plenty of suitably lobotomized synthetic traces with a pair of cheap-ish models. Or even just using good old fashioned NLP to aggressively remove stop words and reduce trace words to lemmas.
today at 6:20 PM
Well done, and great iteration.
The pareto frontier needs clearer distinction. Benchmarks miss half the story. What, if any, capability is lost by the token reduction (for example, was it like super awesome at Golang before and now kind of sucks? that kind of distinction).
today at 6:47 PM
Unfortunately, itās hard to make a chart of that.
today at 9:24 PM
This is cool! But also: am I wrong for thinking āPareto frontierā is some pretty silly/clever marketing jargon? Is this common phrasing for basically saying: test performance per spend on tokens is decent?
today at 9:26 PM
I don't see why? It is a well defined term that existed prior to the recent AI bubble/revolution, and from what I can see they are using it appropriately.
today at 9:31 PM
Fair enough!
today at 7:50 PM
> The problem: thinking models think too much
I see that with Opus 5, it started thinking like crazy in the last few days , I don't think my workflow is that complicated, still it gets into thinking mode and stays there
today at 8:40 PM
The more I learn about Fireworks the more unsavory they seem as a company. I donāt care what the license says, Moonshot has been openly improving, sharing research, and providing weights for the models that make up your entire bottom line, and the moment you can improve them in reciprocal itās closed weights, āthis is our own proprietaryā nonsense? Where are we that China has better open source ethos than America?
today at 8:49 PM
Why is proprietary-licensed software unethical?
Kimi K3 itself isn't FOSS. Speaking of reciprocity: Fireworks is presumably paying Moonshot serious money for the right to do what they are doing here, since Kimi's license[0] excludes commercial inference providers (such as Fireworks) from gratis use. It requires them to: "...enter into a separate agreement with Moonshot AI before using the Software or its derivative works..."
[0] https://huggingface.co/moonshotai/Kimi-K3/blob/main/LICENSE#...
today at 7:41 PM
Aside. I find the "cost per task" charts both useful and uncanny. Is It better a model that takes me to 90% in 1 dollar or one that takes me to 95% in 2 dollars? Or a different model that too scores 90% in 1 dollar? How much will it cost me the last 10% or 5%? At the end of the day, cost to 100% is what matters and the half (90%) backed solution may require more to reach 100% (or not, who knows?)
today at 8:09 PM
> Is It better a model that takes me to 90% in 1 dollar or one that takes me to 95% in 2 dollars?
It's pretty important to understand if your own work domain is one where the last 5% matters. In a lot of day-to-day software engineering tasks, it doesn't, and one can get crazy mileage out of the cheaper models. OTOH, if you are performing novel research, that last 5% may be worth whatever it costs...
today at 9:39 PM
The 90% and 95% are against some blend of tasks meant to be broadly representative. A pricey model seldom fails a problem that cheap models do well, so there's stratification of tasks by difficulty. Someone doing novel research may be in the "hard" 15% of the blend, where P(solution) goes from one third to two thirds.
On the other hand, if it's cheap to tell whether you got a good solution, and you think the 90 and 95% apply to your task blend, then it's almost always worth trying the cheap model first.
today at 6:20 PM
Need this done for DeepSeek, ideally one of the Flash models.
today at 6:48 PM
And GLM. Both Deepseek 4.1 Flash and GLM 5.3 Flash are quote verbose when thinking.
today at 6:24 PM
If you have the compute, I have the expertise.
today at 8:01 PM
Would be nice to include in fire pass.
today at 6:24 PM
Does anybody know if this would be a good model for creative writing?
today at 7:34 PM
> model
> creative
Choose one.
today at 9:38 PM
Are you a bot?
today at 9:55 PM
You really have no sense of irony, do you?
today at 6:42 PM
Do they mean Opus 5.5 or Opus 5?
today at 6:43 PM
The result? Ember-1 set a new Pareto frontier for Bedside Bench across both open and closed models including GPT-5.6 Sol, GPT-6 Astra, and Claude Opus 5 on cost/task.
"Pareto": 8 hits
"Opus 5.5": zero hits
today at 6:51 PM
Obviously this research was done before 6.0 Sol and Opus 5.5 came out. Your point stands that the frontier moves quickly and small gains can be eclipsed quickly.
today at 6:22 PM
On the smaller end, Quen 3.8, while being extraordinarily capable for a small local model, also suffers from extreme thinking. I wonder if the techniques described here generalize to other models too.
today at 6:29 PM
I suspect it might generalize to other large models, but I don't think Qwen3.8 27B is one of them. Kimi K3 is a 2.8 trillion parameter model, and I suspect that is playing a big role in being able to reduce the length of CoT without taking a hit in quality.
That's just vibes, though.
today at 6:19 PM
I don't think the article mentions Pareto frontier enough.
Also, did I miss a memo? Suddenly every article on AI seems to be talking about the Pareto frontier - or have I just not been paying attention?
today at 6:31 PM
I guess they figure "best bang for your buck" comes off a little too colloquial.
today at 8:11 PM
I would really love if we brought back some colloquialisms in this field. Not that long ago most folks in tech would have had pretty blank looks on their faces when someone started talking about the "Pareto frontier"
today at 6:59 PM
Pareto frontier on some benchmark that I am hearing of for the first time.
Kimi K3 with less reasoning tokens isn't exactly exciting either, and particularly so if the license is less open than original Kimi K3.
today at 8:48 PM
Is there a Pareto frontier for the number of times articles mention or don't mention a Pareto frontier.
today at 6:47 PM
They want it to be the best at something. And it's obviously not the absolute smartest. So here we are.
today at 8:17 PM
when everyones fighting to be 'somewhere in the pile' they need some way to advertise they have made progress while not being the best.
today at 6:40 PM
This is really interesting. I think the Fireworks Serverless Training infrastructure they used to develop it is also unique and needed. Except if someone works at one of a handful of the largest labs, it is very difficult to set up or try any sort of training pipeline. The managed training infrastructure makes it available to more people.
today at 6:55 PM
I canāt help but think itās more expensive tinker.
today at 6:21 PM
It looks like it would be similar to GLM 5.3 Flash, had they tested it...
today at 6:40 PM
[dead]
today at 6:19 PM
[dead]
today at 7:13 PM
[dead]