today at 4:26 PM
Today this is the second post I see where the 'traditional' testing methods (unit, e2e, integration, perf, pen, etc) are re-categorized, are put in new perspectives.
The other one is https://alperenkeles.com/posts/vocab-for-testing/
I truly understand our (only) job is to become the oracles of the machine: https://www.osequi.com/manifesto
today at 5:16 PM
I'm betting on something adjacent but flipped. As agents are writing the code, humans are stopping (and getting worse at) reviewing, so the oracle needs to be mechanical. My personal bet: https://github.com/NMitchem/charpente/blob/main/MANIFESTO.md
today at 4:49 PM
Not sure I’m following, are you implying that deterministic simulation testing is a just a new name for a “traditional” test method? I’m not clear because 1. Deterministic simulation testing is not a new concept. (Foundation DB is one of the poster boys for it) 2. All the test methodologies mentioned are a name for a specific pattern and practice of testing, and DST is the same thing.
today at 5:15 PM
This is the future, I'm convinced.
I’ve been building a framework of my own and it’s already found dozens of important and critical bugs on my team at work.
The next problem to solve after this, though, is formalized intent.
A bug is often a bug if the app is not behaving as intended. What we intend is not always so clear, and it will remain an area of the engineer's and product designer's responsibility, because software is complex, and what we want cannot be computed so simply. Defining and understanding intent will remain a "human bottleneck" for many companies.
today at 8:38 PM
What is "formalized intent" supposed to mean?
Isn't the whole advantage of the LLM that it takes plain language intent as its input? Isn't it better to think of what measures you want to move and what safeguards are inviolate?
What is formalized intent? If we take intent to be some mental phenomena, on some weak basis isn't that what natural language already does? It can then be further "formalized" into something like math or coding languages. Which are then already basically "formalized intent"? Maybe you could plot these acts on some spectrum of formalizing, like perhaps natural language gets more "formalized" if you write a spec, akin to some form of pseudocode? Isn't the promise of the new LLM age that instead of being trapped by designers and their "intents" we can compose software at runtime in accordance with said inviolate safeguards? Why ship software "products" if every customer has their own personal LLM? Why not just ship APIs? Why speak of this "human bottleneck" as a necessity rather than something that seems wholly contrived? You don't think whatever software end state your human team stumbles toward through the field of their foggy intents doesn't already exist in the latent space? In a world of LLM clients why not leverage the clients capacity to act on its own intent?
It just strikes me despite your first two paragraphs, the subsequent ones show a profoundly antiquated way of thinking of this problem. I suspect the notion of "intent" is largely unnecessary here. Perhaps you will clarify to me what you mean though that will show I am mistaken, as again I am quite lost as to what you mean by "formalized intent".
today at 5:32 PM
[flagged]
today at 6:43 PM
[dead]
today at 8:21 PM
today at 6:45 PM
Deterministic testing is one thing and it's badly needed.
But I'm totally convinced we're moving to deterministic execution: n various implementations, in different languages, running on different stacks, and a minimal (maybe even hardware) voter taking care of the n-modular redundancy to pick the correct "answer".
0-day Postgres CVE? No problem: one of the implementation gives bogus answers and is detected as broken.
0-day Java? Same thing.
This shall make 0-day in anything but the voters a thing of the past (and the voters shall be minimal and have a tiny attack surface).
The answer can be an API query but we can go much lower: network packets, deterministically created, by n systems and all going to the voter(s). Whatever the "answer" is, it's going to be deterministic and verified from the lowest possible level.
The implementations shall be written from specs, formally verified, etc. But the one thing that'll guarantee any fuck up is caught is going to be n independant implementations of the same specs.
And as soon as one of the implementation/stack return even just one network packet that doesn't match what the other systems do produce: something is fucked up and has to be fixed.
This shall require throwing a huge lot of the non-deterministic bullshit away but we'll see systems doing just that: n-modular redundancy and voters, at the lowest level possible.
I'm 100% sure we're going there (and AI is going to write most of that for us).
today at 8:13 PM
I agree 100%, and I’m definitely experimenting in that direction with my hobby projects.
today at 7:01 PM
interesting idea. any references (long-form) on this?
today at 7:13 PM
[flagged]
today at 4:26 PM
[flagged]