Design principles. (1) Autoregressive decode is memory-bandwidth-bound — each generated token streams every weight byte from HBM — while prefill is compute-bound; nearly every choice below either moves fewer bytes, amortizes bytes across requests, or avoids recomputing them. (2) The optimization target is goodput: maximum tokens/sec/dollar subject to per-tenant SLOs (p99 TTFT, p99 ITL) and a minimum quality bar — never raw throughput. (3) No quantized or kernel-modified artifact reaches a tenant without passing an automated quality regression gate. I validated the core claims at miniature scale on a local harness (Qwen2.5-1.5B, three precisions, llama.cpp) and a two-replica routing PoC; the physics transfer, only the constants change.
Accuracy degradation under quantization is non-uniform: recovery improves with model scale (~99% at 70B+, and MoE models are the most robust), while long-chain reasoning degrades before benchmark scores do, because per-token errors compound over thousands of decode steps. The policy therefore differs per model class rather than being fleet-wide:
| Model class | Weights / Activations | KV cache | Rationale & trade-off |
|---|---|---|---|
| 200B+ MoE | FP8 (W8A8); NVFP4 on B300 | FP8 | Most quantization-robust class; FP8 doubles tensor-core throughput and halves bytes streamed. NVFP4 (E2M1 + E4M3 per-16-value block scales) adds ~2x more capacity per GPU at ~99% recovery — adopted per-model as evals pass. |
| Dense long-context reasoners | FP8; FP4 only eval-gated | FP8 (largest lever at long context) | Reasoning chains are the accuracy canary; FP4 known to mis-sample low-entropy tokens in long traces. Trade throughput for a wider safety margin here. |
| Draft models | INT4/Q4 weight-only | FP16/FP8 | Draft errors are verified away by the target model — they cost acceptance rate, not correctness — so quantize maximally. |
The quality gate is a CI pipeline per artifact: perplexity delta, task evals (GSM8K/code/domain suites), and a long-chain reasoning suite, with per-tenant minimum-quality floors. Calibration data mirrors production traffic distribution. The heterogeneous fleet requires one artifact per hardware target (CUDA/Hopper, CUDA/Blackwell with native FP4, ROCm/CDNA3 FP8) built from a single source checkpoint — the gate runs per artifact, since numerics differ across backends.
Attention is the long-context bottleneck because naïve implementations materialize the O(N²) score matrix in HBM. We standardize on IO-aware fused kernels (FlashAttention-3 class on H200, FA-4 class on B300, composable-kernel/Triton equivalents on CDNA3): tile Q/K/V into SRAM, compute with online softmax, never write scores to HBM — trading recomputation for memory traffic, a strict win for a memory-bound op. On Blackwell, the bottleneck moves — tensor-core FLOPS grew ~2.25x over Hopper while exponential-unit throughput did not — so B300 kernels emulate softmax exponentials on FMA units (the FlashAttention-4 approach) rather than serializing on SFUs.
Each strategy is defined by its communication pattern, which dictates placement given fast intra-node links (NVLink ~900 GB/s, xGMI) and a 25 Gbps (~3 GB/s) inter-node VPC:
| Strategy | Communication | Placement rule | Cost |
|---|---|---|---|
| Tensor (TP) | All-reduce, 2x per layer, on the critical path of every token | Intra-node only (TP ≤ 8) | Best latency; catastrophic over 25 Gbps; diminishing returns with degree |
| Expert (EP) | All-to-all, 2x per MoE layer | Intra-node; wide-EP only over RDMA fabric | Load imbalance → EPLB-style hot-expert replication |
| Pipeline (PP) | Point-to-point activations (small) | The cross-node axis; tolerates 25 Gbps | Pipeline bubbles; adds TTFT |
| Data (DP) | None (independent replicas) | Cross-node scale-out | Full memory per replica; requires cache-aware routing (§2.2) |
Per model class: 200B+ MoE → EP=8 for experts + TP for attention within an 8x slug; FP8/FP4 weights keep it single-node, DP replicas across nodes; PP only if a model exceeds node memory. Dense reasoners → TP=2–8 intra-node; MI300X/MI325X's 192–256 GB HBM is the strategic asset here — models that need TP=8 on H200 fit at TP=2–4, avoiding inter-node traffic entirely. Draft models → single-GPU DP, colocated with their targets. Rejected alternative: cross-node TP "for simplicity of one big pool" — a two-order-of-magnitude bandwidth mismatch that no scheduler can hide.
Disaggregated prefill/decode. Colocated, an arriving long prefill stalls every streaming decode — p99 ITL is held hostage to other tenants' TTFT. We split the fleet into a prefill pool (compute-optimized, large token batches) and a decode pool (bandwidth/KV-capacity-optimized), with KV handoff over RDMA (NIXL-style connectors) — never the 25 Gbps VPC, where transfer would erase the win. Production data (vLLM on GB300, DeepSeek-V3.2) shows disaggregation holding TPOT under 60 ms at batch 256 where colocated serving exceeds 80 ms, at the cost of more hardware for the same nominal throughput — buying SLO stability, i.e., goodput. Where RDMA isn't available, the fallback is chunked prefill on colocated nodes (bounded prefill slices interleaved with decode steps), which removes most ITL interference without any KV transfer. Pool sizing is dynamic (role reassignment on traffic-mix shift), avoiding stranded capacity.
Distributed KV-cache and prefix caching. KV is managed as a tiered, cluster-level resource: GPU HBM (hot) → pinned host DRAM → local NVMe → cluster KV pool over the RDMA fabric (Mooncake/LMCache pattern), content-addressed by prefix-block hash and namespaced per tenant — cross-tenant sharing is disabled because cache-hit timing is an information side-channel in a multi-tenant cloud. This is what breaks latency compounding in agentic loops: each turn re-sends the growing conversation, and a prefix hit converts that prefill into an incremental cost. The critical companion is cache-aware routing — with DP replicas, random/round-robin routing silently destroys hit rates (≈1/N), so the router scores replicas on prefix overlap, queue depth, and load, with sticky-session affinity and load-based override.
Continuous batching runs everywhere: iteration-level admission (a finished sequence's slot is refilled next step), chunked prefill interleaving, and KV-pressure-triggered preemption with recompute-or-swap. The batching depth knob is set per pool from the SLO: deepen batches until p99 ITL approaches the tenant bound, then scale out. Multi-node coordination stays in the router/scheduler layer (llm-d-style) rather than inside engines — engines stay simple and per-node; global intelligence lives where global state lives.
A 200 GB model over the 25 Gbps VPC is ~80 s of pure transfer — prohibitive for scale-to-zero. The mitigation is a hierarchy plus streaming: (1) node OS page cache — free and dominant for warm nodes; (2) local NVMe artifact cache with LRU over model popularity; (3) peer-to-peer fetch from other nodes over the 400G RoCE fabric (~5 s for 200 GB) — the fleet itself is the CDN; (4) object storage as origin only. Loaders stream and begin serving before full load completes (layer-ordered loading), and snapshot/restore of initialized processes (GPU memory pooling for frequently-cycled serverless models) removes framework init from the path. Two multiplicative levers: quantized artifacts halve or quarter the bytes (FP8/FP4 is also a cold-start optimization), and the router biases placement toward nodes whose caches already hold the model — the same cache-aware principle as §2.2 applied to weights.
Three instrumentation layers feed one KPI model:
Guaranteeing KPIs: SLOs are contracts per tenant tier (e.g., p99 TTFT < 500 ms, p99 ITL < 50 ms interactive; relaxed for batch). The autoscaling and admission signal is SLO headroom, not GPU utilization — utilization rewards saturation, which is precisely what violates tails. Benchmarking discipline: Poisson arrivals at production prompt/output-length mixes (never back-to-back sweeps, which overstate throughput ~2x), goodput-under-SLO as the reported number, and continuous canary replays so every kernel, quantization, or engine change lands with a before/after KPI delta attached. KPIs conflict by construction — deeper batching raises tokens/sec/dollar and degrades ITL — so goodput is the single objective that resolves the conflict, and it is the number this endpoint is operated on.