> Speculative decoding is the process of guessing which tokens a model might generate, then validating those guesses.

As a computer engineer, it’s always interesting to see optimizations applied at different levels of the stack.

Speculative execution became pretty popular in the 90s, eventually used in basically every x86 design.

Then in the mid-2000s the Speculator[0] paper brought that concept to distributed systems, which we’re still seeing work on[1][2].

Everything old is new again (:

[0]https://www.cs.princeton.edu/courses/archive/fall07/cos518/p...

[1]https://www.usenix.org/system/files/osdi25-shen-weihai.pdf

[2] https://www.microsoft.com/en-us/research/publication/distrib...

   Inference techniques either move a deployment along the latency–throughput frontier or push the entire frontier out, creating more efficiency to allocate.

• Ifkaluva

  today at 2:54 AM

  The point is to classify them into two kinds. The kind that shifts the frontier is more powerful, since improves capabilities without incurring tradeoffs.

  • philipkiely

    today at 3:20 AM

    I also wrote this as somewhat of a defense of the techniques that don't move the frontier -- there is a lot of value in being able to pick a point on the curve.

this is a nice and concise writeup. what's striking to me is that these techniques really have not changed in /years/. sure, precision has become slightly lower, spec decoding acceptance has gotten slightly better and the complexity of parallelism is trickier with mixture of experts. but no new concepts in a very long time!

the absolute most impactful improvements for inference comes at architecture design time. I firmly believe everyone who cares about impacting model efficiency should look there

• philipkiely

  today at 12:35 AM

  I think the biggest net new recent technique is P/D disaggregation. And that spec dec is very different now especially post DSpark/DFlash.

  But overall yes the fundamentals of LLM performance optimization have been remarkably stable over the last few years.

  • brrrrrm

    today at 1:50 AM

    perhaps its unfair to say this in hindsight, but it's a fairly straightforward application of little's law that's been around for some time

    https://arxiv.org/html/2401.09670v2

• Ozzie-D

  today at 1:37 AM

  [flagged]

The author does not deeply mention that quality/intelligence is a third dimension here in addition to throughput and latency, and the frontier is jagged so quality and intelligence require bespoke benchmarks to evaluate tradeoffs for speed and cost.

• philipkiely

  today at 1:16 AM

  These are both good points that I attempted to cover, quotes:

  > In practice, the efficient frontier is very jagged. Rather than a smooth, continuous line between outcomes, small changes can have big impacts. These cutoff points are often unintuitive and must be discovered empirically through sweeps.

  > However, quantization introduces a new set of tradeoffs between quality and serving efficiency. This is a particularly jagged frontier, where a large degree of improvement to serving efficiency is possible with little-to-no reduction in model quality, especially when using microscaling floating-point number formats like MXFP4 and NVFP4.

  Would appreciate ideas on how to explain in greater depth

good recap on the recent inference techniques!

[flagged]

[dead]

[dead]

[dead]