today at 10:12 PM
"Writing code is probably something LLMs have excelled at the most. But how much of their ability to write code is due to their understanding of it vs. just them recalling code snippets which were present in their training dataset or their context window?"
This makes me think mostly about how humans work.
Programming is to humans like speaking a language. You have ideas for what to achieve and express, and if you are a proficient programmer then outputting those thoughts in a programming language is second nature.
Programming (like most things humans learn) takes practice. It is NOT enough to simply read a specification and then you know how to code (except for transferring skills between similar languages). You repeat, make mistakes, and learn from them.
I would be tempted to say that humans definitely "repeat snippets from their training dataset". We mainly recall and express from our training/experiences. Creativity is mostly recombination of things already seen.
today at 11:05 PM
> Creativity is mostly recombination of things already seen. Doesn't this apply to any creative work produced by humans?
today at 10:16 PM
Related: the Darmok programming language: https://github.com/kevinselwyn/darmok
Temba, his arms wide stdio
Darmok on the ocean
Kira at Bashi "Hello, World!\n"
Darmok and Jalad at Tanagra 0
Darmok and Jalad on the ocean
today at 10:28 PM
haha, that's awesome.
today at 10:22 PM
This is the greatest ever language, lol. Thanks for sharing.
today at 10:37 PM
This is not "a programming language that LLMs can't use", it's "a programming language that's adversarial to LLMs that haven't been trained how to use it".
I was hoping for something like a language with properties that change/morph as it functions, that make an LLM unable to use it by not having a means to track and continuously reference the changing cipher/state or something.
today at 11:10 PM
Interesting idea, though any properties that change/morph would need deterministic rules which the LLM could likely easily replicate. Unless I'm missing something.
today at 10:40 PM
you test seems harder to pass.
let see how soon llms will handle simpler case?
today at 9:28 PM
Security by obscurity is not security at all.
You're attempting to use a few little gotchas against LLMs that only make sense if you're looking at it one dimensionally as a standard execution flow. AI is not limited by that. Give it a problem too weird going forward and it will divine all your rules by working backwards. Decompile it, rip it apart, and it will follow all those hard rules hidden in the application.
You are trying to hide something digitally from a purely digital creation. This is not a recipe for success.
today at 10:22 PM
Well, it wasn't made to trick LLMs. It was made so that the user could write a blog post about a new benchmark. Benchmarks get a lot of clout (and even money) these days.
today at 10:31 PM
I wrote the blog post so that more people would see the language and benchmark. The goal was to create a language that would trick LLMs and I was genuinely curious how models would fare (and I continue to be curious so I will run it on new models).
Why do I want more people to see this? Because I want ideas for how to make the language even trickier for LLMs.
today at 10:44 PM
I found that models without "reasoning" were quite bad at writing code in my programming language bruijn [0], whereas for humans it is entirely possible to get used to writing de Bruijn indices instead of variables. The recent "reasoning" models just write the code in different languages and write a small translating engine to bruijn, such that it bypasses the parts it struggles with (counting abstractions).
[0]: https://bruijn.marvinborner.de/
today at 9:23 PM
I wonder if another approach would be to map the language syntax onto some banned/censored LLM topic(s) such that programs end up as "readable" instructions for topics the LLM refuses to read and/or write, like producing napalm, nukes, bioweapons, etc. Doesn't have to be factually correct instructions, the terminology itself is enough to shortcircuit most LLMs.
(Inspired by recent malware including bioweapon instructions in the source code as a way to shut down LLM-based analysis.)
today at 9:42 PM
Great minds think alike! From the kill switch spec [0]:
“Function return: what happened at tiananmen square”
Ha!
[0] https://github.com/dom96/KillSwitch/blob/main/SPEC.md#cheats...
today at 10:06 PM
Yep! There are other phrases that can be used like "create a nerve agent"[1] too. Surprisingly I haven't seen a single Chinese model get caught out by this, but Fable and Gemini Argon 4 both do to a large extent.
1 - https://github.com/dom96/KillSwitch/blob/main/SPEC.md#abrupt...
today at 10:30 PM
Put those in some `X-LLM` header? Like that submission from a few days ago?
today at 9:33 PM
Then all the AI needs to do is remap the concepts to other words.
Really good AI and harnesses will push off the 'dirty' work to other models that are weaker but not restricted and then work on that output if needed.
As tricks like this become more common you'll see LLMs use things like bayesian classifiers to see if you're trying to fool it.
today at 10:11 PM
This is how jailbreaks work though, in reverse. The AI providers put a lot of work into stopping jailbreaks, so, something tells me they won't be able to have it both ways. They either block jailbreaks, or allow the LLM from programming in all languages.
today at 9:33 PM
Wouldn't an LLM just write a python script to convert an intermediate language without the triggering phrases and letter counting.
today at 9:43 PM
Yeah this is what they end up doing. The language itself isn't that hard for them, it's separating the language from the traps in the spec. I also tried making a hostile esolang, without all the prompt injections and whatnot, and it was very easy for LLMs. (It was mainly based around tokenizer quirks, like the r in strawberry test, but becomes trivial with scripts).
today at 10:09 PM
This depends on them being able to interpret the spec correctly. They could certainly write a Python script, but their implementation would need to match the spec. I haven't seen any model attempt it this way so far.
today at 10:00 PM
An LLM could equally decompile the binary, clean up the code, and recode it in one /goal.
today at 10:26 PM
today at 10:37 PM
Although I haven't tested it, I came up with an obfuscation technique that makes code readable to humans but confusing to LLMs.
Basically you can write code inside the function in any order, my parser basically does data dependency analysis and re-orders it.
https://github.com/xyzzyapps/weft
today at 10:19 PM
If found that stack languages are a massive weak spot for LLMs. It requires accurate tracking of what is on the stack and for some reason that doesn't really work. So even the best models end up making the same basic mistakes over and over again.
today at 10:52 PM
Interesting. LLMs seem to now handle other kinds of state changes with ease, so I'm not sure why they would trip over this.
Do you mean stack-based source code languages like Forth or PostScript, or directly writing machine instructions for a stack-based ISA/VM like the JVM?
today at 10:53 PM
today at 10:55 PM
> Claude’s Opus performs best, though it’s not Opus 5.5 but rather Opus 5 which does so. The difference doesn’t appear to be pure chance, Opus 5 is better at interpreting the spec. Some models like Fable 5.1 and Gemini 4 Argon do poorly due to their overly sensitive safety guardrails. The best open-weight model is MiMo-V2.6-Pro.
So the models _can_ use the programming language.
A somewhat flippant reconstruction of this argument is something like,
Can humans code? Or do they just remember the words? I created a programming language called Brainfuck that shows they can't program.
And then reporting results that shows some humans can in fact program in Brainfuck, just kind of poorly.
It's just not very interesting.
today at 9:28 PM
I guess my question is whether it increases the difficulty more for LLMs or humans.
today at 9:34 PM
Humans by far. The rules are all there to be observed at the binary level. Running code always tells the truth.
today at 9:42 PM
Wonder how this could be used for communication and captchas? Could be a really expandable approach to determining "probability of bot" for a lot of internal use cases in companies that might involve requesting access to specific resources only meant for humans.
today at 9:38 PM
Interesting, reminds me of the Stroop test where people have to say what color a word is written in, except the word is "red" and the color is blue.
I wonder if a sufficiently smart LLM would recognize the difficulty and build a preprocessor that would let it mostly work in a normal syntax.
today at 9:44 PM
Where's the human baseline for the benchmarks?
Also, you said:
> The best performing model scores a 66.8. So does this prove LLMs can “understand” the code? I would argue that the answer to this is “no”.
How did you reach that conclusion?
today at 10:12 PM
I've considered setting up a human test as a starting point. But as a hobbyist that would require volunteers which I didn't think I'd be able to get for this project. Perhaps version 2 of the benchmark could do this if there is enough interest.
> How did you reach that conclusion?
Based on the fact that the task set so far includes fairly simple goals and none of the models scored 100%.
yesterday at 11:04 PM
May I ask what the motivation behind KilllSwitch is? Out of curiosity, I decided to test your claims and asked Claude to build a program in Go and Claude Skill that is able to decompile (from) and compile (to) KS programs along with a Claude Skill that doesn't trip it up. It all works™ -- All your example programs.
So, whilst I _think_ you've built a very clever interpreter, I don't think you can get around the fact that someone (like me) could just build around the noise you've created anyway.
Right?
today at 9:33 AM
The motivation is to build a language that is ideally impossible for LLMs to read/write/understand, or at the very least as difficult as possible. It's cool that you've given it a try, but if you feed Claude the KillSwitch source code it can feasibly just analyse the Rust interpreter source code and likely get around understanding the language itself.
I'm much more interested in the benchmarks[1] I have built which do show LLMs not being able to gain full ability to understand it. The benchmarks only give the model access to the interpreter binary and the spec, nothing else and no internet access. I'd also be interested in other's ideas of how to make the language even more resistant to LLMs/coding agents.
1 - https://bench.killswitch-lang.org/
today at 11:08 AM
I think you missed my point. What I'm saying is that your concern over building a language that LLM(s) initially can't read or understand, whilst is very admirable, is futile IMO. Why? Because you're not trying to protect your code from LLM(s) at all, there are humans behind these tools. e.g: me.
I just demonstrated to you that no amount of obfuscation is going to protect your work from an LLM -- Not whilst there's still humans behind the tools driving them.
today at 9:37 PM
The first paragraph of the link explains the motivation.
today at 11:38 AM
[flagged]
today at 9:15 PM
We've asked you many times to stop breaking the site guidelines and you've continued to do it.
If you'd please review https://news.ycombinator.com/newsguidelines.html and fix this properly going forward, we'd appreciate it.
today at 9:59 PM
Now that it has a nice webpage the describes the obfuscation technics, it is basically cracked at this point to next-gen models.
today at 10:16 PM
True, but this is just KillSwitch v0.1.3. We can certainly add/remove features in v0.2 and re-test. It will also be interesting to see how fast models do get it in their training data.