In the last couple of days I wanted to try out the new definitive DeepSeek v4 releases. I gave it the repository of a semi-abandoned video compression codec and I told it to perform the usual benchmark -> profile -> verify -> research -> improve loop. I specifically chose this codec because the authors include a verifier for the bitstream to make sure you don't break stuff if you want to try your own implementation. I gave the agents access to the compiler's profiler and also Intel's VTune, which has fantastic output. In a couple of hours the LLM generated SSE and AVX implementations of the compression and decompression algorithms that almost doubled performance with a single core. Then I asked it to create a CUDA implementation using NVIDIA's NSIGHT profiler as a guide and it also started doing some good work.

Personally, I believe that LLMs should be treated like an advanced version of Prolog or linear programming: you give the constraints, you have a way of verifying correctness, and you give it a clear goal. If the LLM can verify itself and course-correct you can basically leave it on autopilot

• rrhjm53270

  today at 12:20 PM

  I tried kernel autoreasearch using DeepSeek-V4-Flash as well. It spent about 1-2 hours to complete the FlashAttention optimization job (https://github.com/fengwang/FA5090/tree/main/v7) and cost me only $0.2. I believe we are ready to offload a lot of this kind well-defined constrained optimization problems to AI Agent autoresearch.

• eterm

  today at 12:42 PM

  I did something similar recently with Google's C# protobuf library. I had spotted I was getting CPU bound rather than memory bandwidth bound when doing streaming of uint32 buffers in dotnet gRPC.

  I then asked claude to compare the C#/.NET implementation in the library with the C++ version, and it quickly identified that the C# library was missing a couple of fairly cheap optimisations that were present in the C++ version.

  If I can help get a PR merged, then it'll be by far the biggest impact of any work I've ever done.

  I also compared the Rust version, it had this specific optimisation. The far more popular Tokio/Prost library did not.

  Given appropriate guardrails, LLMs are impossibly fast at iterating to find root causes and specific performance bottlenecks.

  • Almondsetat

    today at 12:59 PM

    You presented another thing LLMs excel at: integrating something from a project that is not present in another one. I think they work so well at this because both the starting and ending points have an already existing structure, so the LLM can guide itself effectively. In your case it's even more egregious because we are talking about the same exact algorithm/functionality implemented in two different, but rather similar, programming languages.

    Could you have manually profiled and compared the execution paths? Sure. Could you have translated the C++ optimizations to C#? Sure. But in such an obvious case, the LLM managed itself.

• _zoltan_

  today at 12:21 PM

  This is exactly how I use it. I mean not on abandoned repos, but in a benchmark - profile - verify - research - improve loop.

Isn't cholesky - used to substitute householder at a point - faster but less stable in some cases? I'm just recalling from memory since I had done a small project on qr decomposition with householder for an exam this year. I mean, if it is faster than the standard torch operation probably there are good reasons for which it is not the default standard torch operation. Might as well be wrong, I'm not sure

Achieving a 232x speedup on a kernel via automated tooling is an incredible engineering feat. Fascinating read on optimization.

Training material seems to be especially rich re GPU kernels and SIMD.

I wonder if there is extra effort put into this because they are useful for the researchers working on the models or just a sub-domain that language models are a great fit for and humans have trouble with?

some of these submissions seem to be omitting the actual rules. the #1 on edinh has a line that says "bypass ban check"

Damn! If a solo engineer can do this, it makes the most around OAI/Anthropic start to look pretty weak.

• dzbarsky

  today at 12:48 PM

  This was nowhere near the top submission. But even if a solo engineer could get a top kernel, you don't think that having thousands of engineers, infinite tokens, and stronger models than are available to the public would give the labs a significant edge?

This is really cool - I really like the beam search idea,

• oinoom

  today at 12:36 PM

  this is the first time ive heard of beam search. i would have reached for a genetic algorithm of some sort, although it seems like some stochastic versions of beam search exist to avoid local minima. i wonder if there are any good frameworks for building these that agents can construct and use.

People are always going to hate auto-research and "loop engineering". Because it's got 2 properties:

1) it's the only way to get something out of models (or people for that matter) that they don't know yet.

2) it's harder to do with an LLM than without. Not easier.

3) and when you fuck it up, half the time the LLM (or other ML technique) makes a fool out of you and you spent $1000 to find the quickest way to get a robot leg on the ground is just to crash it into the ground.

• suddenlybananas

  today at 1:05 PM

  What do you mean by 2?

Every step here has an oracle: wall-clock, the profile, pass or fail from the verifier. I had an agent-built app audited task by task, 10 came back done and 7 worked, and the three misses were the ones needing a credential or a setting on someone else's dashboard. Nothing in the loop could tell the agent it had failed, so it said done and moved on.