one rack of serversinference serverthis GPU (1 of 8)empty slot, no memory ¯\_(ツ)_/¯the model: a 70 GB filecopied into the GPUs' memory at startup2 processors (CPUs)RAM8 GPUsGPU (like an Nvidia H100)processor: does the multiplicationseach stack: 8 memory chipsmemory: 5 stacks, 80 GBthe model lives here: 70 GB of the 80the other 10 GB: mostly the conversations in progress (the “KV cache”)rent: about $3 an hour70 billion weights
860 million → 70 billion
860 million
one token, one layer
70 billion
one token, all 80 layers
first pass: ~300 tokens, mostly hidden instructions and tools
70 GB ÷ 3.35 TB/s ≈ 21 ms per token: at most ~50 tokens a second on one chip
Every token is a full trip through the model.That’s why answers stream, why long answers take longer, and why output costs more than input.