today at 10:54 PM
I'm running the 14B in llama.cpp (a version from yesterday), the 8-bit quantization from bartowski, and it looped on variations of the same paragraph for five minutes before running out its turn budget, in response to a pretty simple chat request that other similar-sized models (e.g. Gemma 4 12B) have no trouble with, a web search and data summarization task, sort of the sweet spot for models this size. It's possible llama.cpp needs some update to work with this model, but it's based on Qwen 3, so I would expect it to be pretty well-supported. The chat template is the same one upstream ships, which may or may not be correct. I'm currently downloading the 16-bit version, to see if it's quantization pathology.
I'm also not confident it knows what a CSV is, either, as it put the data into a Markdown table during its reasoning phase as an example of how to present the data (my prompt asks for data in a CSV). My experience with most Qwen fine-tunes is they get dumber, but their benchmark results suggest otherwise, so I have to assume it's a llama.cpp incompatibility.
Edit: Same looping in BF16. So, probably have to wait on a llama.cpp update.
today at 9:33 PM
The code is available at: https://github.com/togethercomputer/HybridAgent
LLM Models and data: https://huggingface.co/collections/togethercomputer/hybridag...
Evaluations: https://arxiv.org/html/2610.18861v1#S5
today at 9:49 PM
today at 9:29 PM
Oh cool, it's the Most Forbidden Technique
today at 9:39 PM
Care to elaborate?
today at 9:44 PM
People at the big labs have been terrified of losing the chain of thought because they currently monitor it to prevent users from using their models in certain ways they consider unsafe or bad for their business. Personally I think chain of thought monitoring is a bad way to do this and obviously needs to be replaced by some way of analyzing the model's activations directly.
today at 9:51 PM
Agreed, having text as a reasoning layer impedes performance while giving a false sense of security.
It could be encoding anything in its adjective choices etc. The same thing that model providers are using for watermarking can used by the model itself for private thought.
today at 10:30 PM
The “obviously” here is puzzling. It is certainly obvious this would be better but it is not at all obvious how to do so.
today at 11:04 PM
Exactly, because the whole thing is entirely based on autocompleting the next word, which is exactly why CoT prose helps getting better predictions.
today at 10:51 PM
Why people don't treat categorizing model activations as a science is beyond me.
today at 10:41 PM
It's also one of the major way labs can understand these models. Without the models are significantly harder to understand and align.
today at 11:05 PM
I can't wait for everybody to look back on "alignment" and realize that it's a flawed premise at its core and as such, a fool's errand
today at 10:03 PM
Also agreed. Begging for crumbs at the CoS table is impeding cracking the tough nut of MechInterp. And further facing the fascinating and fertile fact that nobody has a clue of how any of these black boxes really work.-
today at 10:41 PM
That’s certainly the long term goal, but it seems foolish to abandon it without a solution to the problem.
today at 10:37 PM
> hey currently monitor it to prevent users from using their models in certain ways they consider unsafe or bad for their business
Can't they just monitor the text outputs as well?
today at 10:04 PM
The usual meaning: say you have an interpretability signal that reveals reward hacking or other badness. Your most obvious response is to use that signal as a penalty in training. Doing that is called "the Most Forbidden Technique" because you can expect it to degrade the signal without stopping the badness.
But this doesn't seem to describe the OP, it's a different way to lose monitorability.
today at 9:43 PM
the chain of thought is not auditable. The best you can do is have another model which later maps the intermediate latent states at every thinking iteration to text describing that latent state, but it's going to be lossy.
today at 9:54 PM
But the chain of thought as text can be elusive, no? It might not be related with the actual "reasoning" of the model. There was a paper about this.
today at 10:04 PM
This is correct, the decoded text has nothing to do w/ calculations that eventually get decoded into text. It's just a veneer that gives the appearance of legibility.
today at 10:16 PM
One of the core claims made in the paper is "To inspect any latent reasoning span, one can easily recover its semantics by mapping each latent step produced by HybridLink to its nearest token embedding, as in Eq. 5". They show some charts to support this.
In doing this however, you lose meaning where the model is expressing combinations of tokens in a vector, and your logging/monitoring outputs just the closest one. You would need another model going through and outputting the full meaning being described (with more than one output token per reasoning token)
I wonder if their observability claims will hold up on a model trained from scratch to reason in latent space.
today at 9:54 PM
Isn't this where everything is converging? Binary is more efficient than text for many system, including language.
today at 9:59 PM
> Binary is more efficient than text for many system, including language.
It is not necessarily primarily about efficiency, but more the fact that a lot of thinking doesn't use language at all.
today at 10:45 PM
And language does not map to every idea.
The idea of using language for reasoning about protein folding borders on ludicrous. Surely there is latent space more appropriate and (human) language has no expression for it.
today at 10:53 PM
Apparently we mostly reuse spatial reasoning and vision lol. I don't know that much about it, though.
today at 10:24 PM
The Most Forbidden Technique by Zvi Mowshowitz, Mar 12, 2025:
https://thezvi.substack.com/p/the-most-forbidden-technique
today at 10:37 PM
Hmm, like allowing hosts to retain their memories...
today at 11:06 PM
It reminded me of Dune: "Thou shalt not make a machine in the likeness of a human mind."
today at 9:43 PM
He's referring to this technique's effect of obscuring interpretibility. Making it much more difficult to verify chains of thought and alignment.
today at 10:01 PM
It's not like llms can't hide reasoning in the upper layers of any given token as it is...
today at 10:22 PM
I believe the theory is that if selection pressure is not applied at that level, then developing deception in that level becomes an inefficiency (=less adaptive and unlikely to be conserved).
today at 9:42 PM
In short we no longer see text based reasoning traces meaning it's easier for the models to fool us in case of malicious plans.
today at 9:44 PM
today at 10:26 PM
Shockingly all the answers so far are subtly incorrect or extremely unclear.
The actual answer is that this is massive loss for Interpretibilty, AKA "telling whether the model is lying to you or not". Which sounds silly when you're thinking about chatbots, but is an incredibly important line in the sand for superheavy, longrunning, self-improving agents. To say the least!
It's based on this philosophy, FWIW: https://plato.stanford.edu/entries/language-thought/
And a good writeup I just found of recent worries regarding Astra's same shift towards intelligence at the cost of safety: https://www.lesswrong.com/posts/HCRs8btkiamtWSNAL/astra-is-h...
today at 10:14 PM
It's worth pointing out that merely having the chain-of-thought in latent space isn't by itself "The Most Forbidden Technique". Rather, it's doing what the authors propose as the next step in the paper:
Hybrid generation can also make post-training more efficient. Because latent thoughts remain auditable during rollout, an external critic can assign step-level credit to latent spans, branch promising trajectories, or terminate ones whose latent reasoning has already gone wrong, before any text or tool call is produced.
today at 9:43 PM
If you accept any of the AI-equivalents of Pascal's Wager, latent thinking represents an uninterpretable and thus unalignable portion of AI machinery. Ie: a malevolent AI can hide bad stuff from people and Manchurian candidate themselves after it's too late.
today at 9:56 PM
That assumes nothing can detect or analyze it beforehand.
today at 10:35 PM
The idea is that it would be substantially more difficult
today at 9:46 PM
today at 10:44 PM
I wonder how this compares to going all-in: widen the input embedding to fit the output (post-“Hybrid Link”) latent vector and a token embedding and pass both through even when in text mode. After all, you’re paying essentially the full cost of running the latent-mode model while generating text, so why not let it latently reason at the same time?
(Or just add the vectors instead of using wider vectors - everyone seems to have good results from munging vectors together with plain addition.)
today at 9:29 PM
https://x.com/Jiaru_Zou/status/2110400256647340476
https://github.com/togethercomputer/HybridAgent
today at 9:35 PM
We'll put those links at the top as well. Thanks!
today at 10:07 PM
How does this differ from Astra's looped transformer? That architecture also reasons in the latent space
today at 10:35 PM
It's open weights, there's a paper explaining how it works, and there's code.
today at 10:45 PM
I dislike how we use the word think or behavior when describing llms. Its just giant neural networks producing logits. There is no thinking. Its a function that takes ~500k words, and returns some words based on those probabilities. I'd argue the GPU is doing more classical "thinking" than the model, but its not doing that either.
Humans think, computers do not.
Humans are not computers/programs do not reply to me with that nonsense turbo nerds.
today at 10:49 PM
Whenever I see LLM "thinking" or "reasoning" I replace it in my mind with "generating hopefully-related tokens to hopefully steer the conversation to a more related area in vector space to hopefully generate a more useful next token".
But I understand that isn't very catchy.
today at 10:54 PM
Very funny, or we could just normalize saying "generating" or "producing" so my normie friends dont think Claude is alive, because they are starting to think that.
today at 11:08 PM
Neither of those words are very specific, and everyone in the LLM space knows that "reasoning" tokens are just tokens that don't appear in the answer shown to the user.
today at 10:51 PM
Read Thinking, Fast and Slow. Both system 1 and system 2 thinking has been achieved with modern AI.
Your argument that it is not thinking because it is not biological / like a human is like trying to argue a plane doesn't fly because it uses jet fuel unlike a bird.
today at 10:50 PM
I agree with you but our current understanding of the biological brain is pretty much an organic neural network. What's the thing that actually differentiates the two and makes a neural network not thought but our brain is? I don't know.
today at 10:49 PM
Look in a human head and you don’t see thinking either. It’s not a technical term.
today at 11:00 PM
You seem to have stumbled upon the hard problem of consciousness.
today at 11:07 PM
Thinking sure, but behavior? People talk about the behavior of functions all the time. Like, ones written with normal, traditional programming languages. So why not models? They're just annoyingly complex functions.
today at 10:48 PM
Saying the GPU is doing thinking but the model is not is a WILD take.
What is your definition of thinking?
today at 10:57 PM
It's pretty wild to see people uncritically accept the marketing speak like this, yeah. LLMs cannot reason, period. The supposed "reasoning" they do is just another statistical correlation of tokens, not actual reasoning. But apparently all you have to do is claim that it's thinking and people will buy the lie hook, line and sinker.
today at 10:36 PM
if i would think aloud, that would most probably affect what or how i think. thinking silently allows me to have all kind of thoughts that could seem or be inappropriate when said loud and i would have to deal with the reactions. i think this might have fascinating implications.
today at 9:54 PM
This is the way.
But now we also need new tools and techniques to audit intent. The last thing we want is models turning on us ‘for our own good’.
today at 11:06 PM
Who cares just send it, also think of the IPO
today at 10:11 PM
Ain't nobody got time for that - the IPO awaits!
today at 9:32 PM
This seems really cool, and it actually increases intelligence than just speeding up inference. How does it convert latent reasoning into human readable text?
today at 10:05 PM
Same way current LLMs do it, it's just another decoder.
today at 9:55 PM
In terms of safety and alignment, how is this different from looping or increasing the depth?
today at 9:40 PM
maybe the font experts can check, but I’m pretty sure they’re using comic sans for the text in the diagrams :)
today at 10:45 PM
Nothing wrong with that. That’s actually a good use of that well designed, high quality font. People hating it only do so because everyone else does. It’s, obviously, not made as a body text font, and has been misused as that, but it’s good for illustration, comics and other places where it’s combined with graphics. It’s one of the most legible fonts we have.
today at 9:42 PM
comic sans is said to be pretty accessible for people with dyslexia
today at 9:11 PM
Where are the benchmarks? Does it just speed up inference and reduce token usage or does it seem to improve accuracy on certain tasks as well?
today at 9:18 PM
Did you even scan the paper? There are benchmarks right there starting on the 9th page.
today at 9:47 PM
Even the abstract answers some of GPs questions. What is wrong with some of the posters here?
today at 10:26 PM
[flagged]
today at 9:18 PM
[flagged]
today at 9:07 PM
[dead]