> I note the lack of performance information. I can only imagine it's much, much, slower than any other way to run a larger model (including, e.g. using system RAM and streaming some stuff from disk)
Not necessarily, and I suspect there are plenty of configuration for which this isn't going to be the case. Let me explain why:
- when offloading the weights to RAM or NVMe, you need to transfer the massive weights from your slow storage to the GPU for each layer being processed for each token. And as such you are being bottlenecked by the transfer bandwidth (which is either the men bandwidth of your DRAM or the read speed of your disk)
- when using a distributed setup, the weights stay in the VRAM on each machine, the it's the GPU memory bandwidth that matters for the weights, and it's much higher than the two other bandwidth discussed above and as such the bottleneck isn't here. You need to tranfert data from a group of layers sitting on one device to the next one another device, but the amount of data is much smaller than the weights (we're talking about kilobytes of data, not gigabytes) so the network throughput isn't a limiting factor.
The limiting factor is the network latency: if you split your model between 4 devices, you'll have 3 times the network latency per token. If you're on a network with 1ms latency, that means 3ms of latency per token. Which means the theoretical upper bound for your inference speed without speculative decoding is 30tps (this theoretical limit assumes the computation itself is instantaneous).
So this is unlikely to be practical over the internet (too high of a latency) but on a local/enterprise network with speculative decoding it could totally work.
Edit: note that all of the above is about token generation, for prefill/prompt-processing the distributed setup will almost certainly win (because in this case, the network latency doesn't add up)
chromatin
today at 1:47 PM
A 1 gbe local network should have < 1 msec latency per hop so theoretical upper bound is substantially higher than 30 tps (again assuming instantaneous compute) => thus network latency should not be the limiting factor in reality, no?
On a fast network, yes. If you want to create a distributed network over the internet, then it will.
I’m staring at this comment for a while now: With 3ms latency combined per token, wouldn’t that mean (1 / latency) = 333 token/s for the theoretical upper bound? I’m not trying to nitpick, just curious if I misunderstand something.
Indeed, I completely screwed my math up. Looks like 10am is too early in the morning for a Sunday.
33 tps max token generation speed would be for 10ms of network latency in the above example.
Ah, that's interesting. I though there was more data crossing the network. So, why does a DGX Spark come with super fast network if 10Gbps ethernet would be sufficient for splitting a model? I never bought a second Strix Halo on the assumption that the pipe between them would be a limiting factor to using larger models, so obviously there's something I don't understand.
The amount of data is only low for inference, not for training, and AFAIK DGX spark is supposed to be a researcher's machine that can do small-scale training.