Although I'm interested in both topics (KV compression and attempts to stream MoE models from storage) this is at least the 10th vibecoded project on this topic I've seen today alone across HN, Twitter, and some subreddits I visit.

At least this one gave credit to the upstream projects which it used as a reference.

The llama.cpp project is also getting a wave of vibecoded PRs that are very clearly being produced by pointing claude at the repo and the original paper and having it produce something.

Almost none of these attempts contain information that really matters, like actual benchmark tests with differen KV quantization levels (not just perplexity or KLD).

• _zoltan_

  today at 7:15 PM

  "vibe coded" is NOT the bad thing you think it is.

  Going from paper to implementation from scratch in half an hour or so is great.

  • mjr00

    today at 7:21 PM

    > "vibe coded" is NOT the bad thing you think it is.

    It's not inherently bad in the same way that a first draft of a novel is not inherently bad.

    But if someone asked me to read their novel and it was a first draft that they themselves had clearly not bothered reading or editing, I'd tell them to fuck off.

    • sumeno

      today at 8:08 PM

      At least in the novel example the author had the decency to write what they're asking you to read.

      These are more like sending someone who didn't ask you a question a LMGTFY link they didn't ask for and expecting them to read all the results. Just a complete lack of awareness and respect for the maintainers

  • simonw

    today at 7:58 PM

    Sure, but the problem is when you take that half hour of work and share it with other people without making clear how much effort has gone into it.

    Software is valuable if it has been tested and exercised properly by other people. I don't care if you vide coded it provided you then put the real work in to verify that it actually works correctly - and then include the proof that you've done that when you start widely sharing it with the world.

    Right now it's impossible to tell which of these projects implementing the paper are worth spending time with.

  • pqtyw

    today at 8:07 PM

    If there is nothing valuable it contributes, though? i.e. its not a novel paper then only value is the whatever you personally learn from it.

  • brokencode

    today at 7:22 PM

    That’s a starting spot, but how about some testing and benchmarks?

    Where’s the value added if the person just tells Claude to do it and then submits a PR?

    The maintainers may as well vibe code it themselves if that’s all the work the would-be contributor is going to put into it.

    • yieldcrv

      today at 7:30 PM

      if it works it works

      we live in a wholly unoptimized world because the available resources have been so high, while the benefits of optimizing have been so low. that has flipped now and there are tons of low hanging fruit to optimize.

      I agree that benchmarks would be great, but thats only relevant to this one topic, not the overall agentic coded pull request concept itself

      • pqtyw

        today at 8:08 PM

        It might work, but what's the point is sharing it if anyone can do the same in those 30 minutes with minimal effort?

      • sumeno

        today at 8:04 PM

        > if it works it works

        If it works in one case that doesn't mean it works consistently or well in the general case

        I've made lots of things with Claude Code that just work... until I do things in a slightly different order and the whole thing explodes

      • jmalicki

        today at 7:35 PM

        It's relevant in that it's an example that people are doing the easy part - the coding - and skipping the hard part - the benchmarking and proving it works and provides value.

        A PR without evidence it works and expectations for the benefits using the new feature would bring is kind of worthless.

  • sroussey

    today at 7:38 PM

    The authors of the project have CC as well, so doing this is just eating their time.

  • aegis_camera

    today at 7:44 PM

    Yes, this took time to test :)

I couldn't get the downloadable binary to work, or the binary I compiled myself:

  ./SwiftLM \
    --model mlx-community/Qwen3.5-122B-A10B-4bit \
    --stream-experts \
    --port 5413

  [SwiftLM] Loading model: mlx-community/Qwen3.5-122B-A10B-4bit
  [SwiftLM] Enabled Async SSD Streaming on directory: e9c67b08899964be5fdd069bb1b4bc8907fe68f5
  [SwiftLM]  Memory strategy: FULL GPU (69.6GB model, 133.4GB available)
  [SwiftLM] Download: [===================>] 100% ⠋ (66395.4 MB / 66395.4 MB) | Speed: 0.0 MB/s      
  MLX error: Failed to load the default metallib. library not found library not found library not found library not found  at /Users/runner/work/SwiftLM/SwiftLM/LocalPackages/mlx-swift/Source/Cmlx/mlx-c/mlx/c/stream.cpp:115

• aegis_camera

  today at 8:04 PM

  Let me check, I had seen metallib error during development, let me check.

Feels 100% vibe coded in a bad way.

Llama.cpp already has KV compression and one of the turbo quant PRs will get merged at some point.

If you don’t care about the fancy 3 bit, the q8 KV compression is good enough! Don’t bother with q4

./build/bin/llama-server -m model.gguf \ --cache-type-k q8_0 \ --cache-type-v q8_0 \ -c 65536

Etc

We implemented two techniques to run massive 100B+ parameter MoE models natively on the M5 Pro 64GB MacBook Pro:

TurboQuant KV compression: We ported the V3 Lloyd-Max codebooks from the TurboQuant paper (Zandieh et al., ICLR 2026) into native C++ and fused dequantization into Metal shaders. This achieves a measured 4.3× KV cache compression at runtime, completely eliminating Python overhead.

SSD Expert Streaming: To fit a 122B parameter model (e.g., Qwen3.5-122B MoE) without triggering macOS VM swapping or Watchdog kernel kills, the full ~60 GB weight file remains on NVMe. Only the top-k active expert pages are streamed to the GPU per forward pass at ~9 GB/s. As a result, inference runs with only 2,694 MB of active GPU VRAM on the M5 Pro 64GB, while the OS page cache automatically handles hot-expert reuse.

By combining these two approaches, we can comfortably run massive models in memory-constrained environments on Apple Silicon.

Also tested QWEN 4B on IPHONE 13 Pro.

Code and implementation details: https://github.com/SharpAI/SwiftLM

• anemll

  today at 8:03 PM

  Check it out, you might be able to speed it up using this https://github.com/Anemll/anemll-flash-mlx https://x.com/anemll/status/2038684375425200360

• altruios

  today at 7:07 PM

  what tokens/s are you getting with a 122B MoE model in this setup? I didn't see any benchmarks in the benchmarks section on the readme.md

  • aegis_camera

    today at 7:48 PM

    I'll add more details. We just wired up the pipeline on both MAC and IOS.

  • gigatexal

    today at 7:29 PM

    yeah this I'd like to see added to teh readme.

I like this idea on expert streaming. I've been poking around fairly thoroughly at the same idea - can we fix a set of experts? when can we fix them? How long is the top-k selection "good" for in terms of number of forward passes?

One thing I've turned up in smaller models and I'm sort of winding my way toward verifying in larger ones is that if you train the MoE model from scratch with this kind of knockout / subset of experts baked in, then you get significantly better loss outcomes. In small models, it's actually better than training an MOE without conditioning on a reduced set of experts per pass.

Anyway, pretty cool. There's some Pareto-optimal curve based on memory bandwidth, amount of GPU / unified RAM and inference compute times for streaming stuff in.

• aegis_camera

  today at 8:00 PM

  [dead]

Does this use anything from the flash-moe project?

https://github.com/Alexintosh/flash-moe

• aegis_camera

  today at 7:52 PM

  Yes, this is a reference project, the main different is we don't use os swap ( it introduces latency, will add https://github.com/danveloper/flash-moe to the original reference as well ).

Another project without running real benchmarks. It's very easy to generate tokens, it's much harder to solve tasks locally.