It's a reasonable summary, although I feel there's some fundamentals missing. I wonder how much it will make sense to a platform engineer who doesn't already have a very solid understanding of transformer architecture internals such as the Attention mechanism and its more recent variants. I don't know if you can really get this stuff without having some underlying model internals picture in your head of what the KV cache actually is and how it falls out of repeated attention computations on a continuously growing prefix, to say nothing of understanding KV cache optimizations like PagedAttention which depend very much on the details of how you compute the attention weights with matrix multiplications, softmax, etc. I certainly couldn't make sense of something like "Rather than recompute that history, the engine keeps the attention keys and values for every token in flight" without a pretty solid understanding of how Attention works, which I had to gain from other sources.

This is also a shortcoming of Philip Kiely’s otherwise fantastic book Inference Engineering; it kind of glosses over the transformer architecture details, which you need to get elsewhere. Sebastian Raschka's books are very good for that but you have to then link it all back to how to do efficient inference, which he doesn't talk about so much.

• d-ramdass

  today at 8:44 PM

  That's fair. We tried to balance level of detail to make it approachable for newcomers to the space as well as practical tips for experienced practitioners. That being said, we'd love to incorporate your feedback in the guide - feel free to contribute to the repo and we'll also add those topics to our list to cover

It looks rather generic. And to be honest I was able to use an agent to hand roll cluster serving stack using vllm, completed with spec decode, kernel cache reuse, and fixing or forward porting engine bugs.

• d-ramdass

  today at 9:27 PM

  You can definitely use an agent to help Day 0 deployment but we found this information useful while operating a platform with SLAs for real users (see this series - https://www.upbound.io/blog/drinking-our-own-champagne-part-...).

  Feel free to help contribute to the guide also!

  • randbyte

    today at 9:45 PM

    It’s a 32 node gb200 cluster across 3 az (with one sitting in house), running 3 different open weight models. The remote az are subjected to preempt all the time too. So it’s much more than a day 0.

    My point is that no one should actually read a guide like this and rollout the stack manually. A bootstrap agent can do all these by reading the engine-model cookbooks and combine with distributed system common practices (which the models know quite well).

    The “cutting edge” issues such as engine bugs for new model releases, topology and spec decode conflicts, are what the models don’t necessarily directly know and needs human collaborations. They are very situational and is not covered anywhere (and shouldn’t be).

This guide is for platform engineers new to inference. If you need to understand how models and GPUs work together and how to run inference in production, this guide will help. You’ll also learn about the tools to use and how to grow into an inference engineering role. This guide was written by the maintainers of the open source CNCF sandbox project Modelplane

[flagged]

[dead]