DigitalOcean — Agentic Inference Cloud · Full Specification Deck
Production Inference
Optimization Strategy
All three pillars, in depth · measured evidence throughout ·
strategy mapping, roadmap, and code-review appendix.
Pillar 1
Kernel & precision engineering
Pillar 2
Distributed inference & orchestration
Pillar 3
Resiliency & observability
This is the complete deck: spec depth in presentation form. Slides 2–14 are the deliverable; 15+ are the code-review appendix.
Design principles
Three rules govern every decision that follows
1 · Decode is memory-bound
Every generated token streams all weight bytes from HBM (H200 ~4.8 TB/s, MI300X ~5.3 TB/s). Prefill is compute-bound. Every optimization: move fewer bytes, amortize bytes, or don't recompute them.
2 · Optimize goodput, not throughput
Target = max tokens/sec/dollar subject to per-tenant SLOs (p99 TTFT, p99 ITL) and a quality floor. Raw throughput rewards exactly the saturation that violates tails.
3 · Quality gates, not vibes
No quantized or kernel-modified artifact reaches a tenant without an automated regression gate: perplexity + task evals + long-chain reasoning suite.
Method
Core claims validated at miniature scale: 3-precision benchmark harness + 2-replica routing PoC (llama.cpp, Qwen2.5-1.5B, Apple Silicon). Physics transfer; constants change.
These three principles are the answer to "how do you make trade-offs" — everything after this slide is their application.
Pillar 1 · Precision management
Quantize by model class, gate by evals
| Model class | Weights/Acts | KV cache | Rationale & trade-off |
|---|
| 200B+ MoE | FP8 W8A8 · NVFP4 on B300 | FP8 | Most quantization-robust class (~99% recovery at scale); FP8 = 2x tensor throughput + half the bytes; NVFP4 earns ~2x more capacity per-model as evals pass |
| Dense long-context reasoners | FP8 · FP4 eval-gated only | FP8 — biggest lever at long context | Reasoning chains degrade first (errors compound over thousands of steps; FP4 mis-samples low-entropy tokens) — trade throughput for safety margin |
| Draft models | INT4 weight-only | FP16/FP8 | Draft errors are verified away by the target — they cost acceptance rate, not correctness |
Why non-uniform: accuracy recovery improves with model scale; benchmarks (MMLU) move last, long-chain reasoning moves first. One policy per class, never fleet-wide.
Heterogeneous fleet: one source checkpoint → per-hardware artifacts (CUDA/Hopper, CUDA/Blackwell native FP4, ROCm/CDNA3 FP8). The eval gate runs per artifact — numerics differ across backends. Calibration data mirrors production traffic.
FP8 is the default; FP4 is earned per-model. KV quantization matters as much as weights once context grows.
Pillar 1 · Compute & memory access (SRAM vs HBM)
IO-aware kernels: spend FLOPs to save bytes
- Attention: FlashAttention-3/4-class fused kernels — tile Q/K/V into SRAM, online softmax, never materialize the O(N²) score matrix in HBM. Recomputation traded for memory traffic: a strict win for a memory-bound op.
- Blackwell twist: tensor cores grew ~2.25x over Hopper, exponential units didn't — softmax becomes the bottleneck. B300 kernels emulate exp() on FMA units (FA-4 approach).
- MoE: fused dispatch/expert/combine with DeepEP-style all-to-all overlapped with expert compute; FP8 grouped GEMMs.
- Decode path: CUDA-graph-captured steps (kill launch overhead), paged KV block layout tuned per architecture.
Long-context roadmap: chunked prefill bounds per-iteration compute → FP8 KV halves capacity + read bandwidth → DSA-style trainable sparse attention (lightweight indexer, top-k selection, O(L·k)) as the 500K+ endgame.
TTFT at 2,000 words is quantization-invariant (0.43–0.45s across FP16/Q8/Q4) while decode differs 2.3x — prefill compute-bound, decode bandwidth-bound, phases deserve different optimizations.
Roofline logic: a kernel is memory-bound until arithmetic intensity beats the hardware FLOPS/bandwidth ratio (~400 FLOPs/byte on H200). Decode never gets close — hence every byte saved is throughput earned.
Pillar 2 · Parallelism strategy
The interconnect decides the parallelism, not the model
| Strategy | Communication | Placement (NVLink/xGMI in-node · 25Gbps VPC across) | Cost |
|---|
| Tensor (TP) | All-reduce 2x/layer, on every token's critical path | Intra-node only (TP ≤ 8) | Best latency; catastrophic over 25Gbps |
| Expert (EP) | All-to-all 2x/MoE layer | Intra-node; wide-EP only over RDMA fabric | Hot experts → EPLB-style replication |
| Pipeline (PP) | Point-to-point activations (small) | The cross-node axis; tolerates 25Gbps | Bubbles; adds TTFT |
| Data (DP) | None — replicas | Cross-node scale-out | Full memory each; needs cache-aware routing |
Per class: 200B+ MoE → EP=8 + TP for attention inside an 8x slug, FP8/FP4 keeps it single-node, DP across nodes. Dense reasoners → TP 2–8 intra-node; MI300X/MI325X's 192–256GB HBM is the strategic asset — TP=8-on-H200 models fit at TP=2–4, avoiding inter-node traffic. Drafts → single-GPU DP colocated with targets.
Rejected alternative: cross-node TP "for one big simple pool" — all-reduce on every token over a link with ~300x less bandwidth than NVLink. No scheduler hides a two-order-of-magnitude mismatch. PP is the only cross-node model-splitting axis.
One sentence per row, then land on the MI300X memory point — it's the fleet-specific insight, not generic textbook.
Pillar 2 · Efficiency primitives 1/2
Disaggregated prefill/decode: isolate the two physics
- The problem: colocated, one tenant's long prefill stalls every streaming decode — p99 ITL held hostage to others' TTFT.
- The split: prefill pool (compute-optimized, big token batches) + decode pool (bandwidth/KV-capacity-optimized); KV handoff over RDMA (NIXL-style) — never the 25Gbps VPC, where transfer erases the win.
- Dynamic roles: pool sizing follows traffic mix; static assignment strands capacity when the prompt/output ratio shifts.
- No-RDMA fallback: chunked prefill on colocated nodes — bounded prefill slices interleaved between decode steps; most of the ITL isolation, zero transfer cost.
Production evidence (vLLM, GB300, DeepSeek-V3.2): at batch 256, disaggregated serving holds TPOT < 60ms where colocated exceeds 80ms — and the latency curve stays flat as load grows.
The honest trade-off: per-GPU nominal throughput can be lower. You spend hardware to buy SLO stability — i.e., goodput. That's the correct trade for a multi-tenant cloud selling latency guarantees.
Frame it as buying predictability: the metric that improves is the tail, and SLOs are written against the tail.
Pillar 2 · Efficiency primitives 2/2
KV-cache as a cluster resource + cache-aware routing
- Tiering: GPU HBM (hot) → pinned host DRAM → local NVMe → cluster KV pool over RDMA (Mooncake/LMCache pattern), content-addressed by prefix-block hash.
- Multi-tenant security: caches namespaced per tenant — cache-hit timing is an information side-channel; cross-tenant sharing is off by design.
- Agentic loops: each turn re-sends the growing conversation; a prefix hit converts that prefill to incremental cost — this is what breaks latency compounding.
- Routing: with DP replicas, round-robin destroys hit rates (≈1/N). Router scores replicas on prefix overlap + queue depth + load; sticky sessions with load-based override.
- Continuous batching everywhere: iteration-level admission, chunked prefill interleave, KV-pressure preemption. Batch depth set from SLO headroom, per pool.
2 replicas, 8 concurrent multi-turn sessions: round-robin → cache-aware routing = turns-2+ mean TTFT 110→34ms (3.2x), p95 260→36ms (~7x). Policy change only. Matches Google GKE Inference Gateway production report (35% TTFT, 2x p95) and llm-d's endpoint-picker design.
Once every engine has prefix caching, the load balancer becomes part of the inference stack. My 2-replica PoC is round-robin's best case — production penalty (1/N hit rate) is worse.
Pillar 3 · Cold start mitigation
Make the bytes already there — or cheap to move
The math: 200GB over the 25Gbps VPC ≈ 80s of pure transfer. Over the 400G RoCE fabric ≈ 5s. Prohibitive for scale-to-zero unless the hierarchy does the work:
- Tier 1: node OS page cache — free, dominant on warm nodes.
- Tier 2: local NVMe artifact cache, LRU by model popularity.
- Tier 3: peer-to-peer fetch from other fleet nodes over RoCE — the fleet is the CDN.
- Tier 4: object storage as origin only.
Plus: streaming loaders (serve-while-loading, layer-ordered), snapshot/restore of initialized processes for frequently-cycled serverless models.
Two multiplicative levers:
1 · Quantized artifacts halve/quarter the bytes — FP8/FP4 is also a cold-start optimization.
2 · Weight-aware placement: the router biases new sessions toward nodes whose caches already hold the model — same cache-aware principle as KV routing, applied to weights.
Identical model load: 7.4s from cold disk vs 0.6s from warm page cache — 12x from cache placement alone, before any exotic engineering.
Answer the "100GB+ models" ask with the bandwidth math first, then show the hierarchy exists to keep loads off the slow link entirely.
Pillar 3 · Telemetry & benchmarking
Three layers of instrumentation, one KPI model
| Layer | What's captured | Cadence |
|---|
| Kernel | Nsight / rocprof traces; roofline placement per kernel; attention vs GEMM vs comms split; catches "kernel fell off the fused path" regressions | Sampled, canary replicas |
| Engine | Per-request streaming timestamps → TTFT (decomposed: queue + prefill), TPOT, full ITL distribution; queue depth, batch occupancy, prefix-hit rate, preemptions, KV utilization | Always-on |
| Fleet | Tokens/sec/dollar per model/hardware/tenant (true cost incl. idle amortization); SLO burn rate; routing quality | Always-on |
Guaranteeing KPIs: SLOs are per-tenant-tier contracts (e.g. p99 TTFT < 500ms, p99 ITL < 50ms interactive). Autoscaling/admission signal = SLO headroom, not GPU utilization — utilization rewards the saturation that violates tails.
Benchmark discipline: Poisson arrivals at production length mixes (back-to-back sweeps overstate ~2x); goodput-under-SLO is the reported number; continuous canary replay attaches a before/after KPI delta to every kernel/quant/engine change.
TPOT is a mean; ITL is a distribution — users feel the p99 stalls that TPOT hides. That's why ITL is a first-class KPI and why disaggregation exists.
Evidence · Benchmark harness
The measured trade-off surface (all four KPIs)
| Precision | Size | Decode tok/s | TPOT | ITL p95 | TTFT @2000w | Quality gate |
|---|
| FP16 | 3.56 GB | 75.8 | 13.2 ms | 13.8 ms | 0.426 s | 6/8 |
| Q8_0 | 1.89 GB | 119.0 | 8.4 ms | 9.0 ms | 0.444 s | 6/8 |
| Q4_K_M | 1.12 GB | 176.5 | 5.7 ms | 6.0 ms | 0.453 s | 6/8 |
Same two questions missed at every precision — a capability limit, not a quantization artifact. This is what "eval-gated" looks like in practice.
Strategy
2026 techniques mapped onto the AI-Native Cloud
| Where it lands | Technique (source) | Customer impact |
|---|
| Inference Router (preview) | Prefix-cache-aware endpoint picking (llm-d EPP pattern) | 3–7x multi-turn TTFT for agentic tenants — software-only; my PoC + Google prod numbers |
| Serverless fleet | Cluster KV pool over the 400G RoCE fabric (Mooncake/LMCache) | Any replica reuses any prefix; sessions survive rebalancing; higher effective batch |
| Inference Optimized Image | EAGLE-3 / MTP speculative decoding + FP8 KV defaults | 2–3x TPOT on chat/code; MTP heads come free with DeepSeek-class models |
| MoE serving (DeepSeek, Qwen 397B) | Wide-EP with DeepEP/EPLB dispatch + P/D disaggregation over RDMA | DeepSeek's stack sustains 73.7k in / 14.8k out tok/s per H800 node — the margin exists |
| B300 fleet | NVFP4 artifacts, eval-gated per model | ~3.5x weight memory vs FP16 at ~99% recovery (large/MoE) → more models per GPU |
| Scale-to-zero serverless | Hierarchical weight caching + streaming loaders | Cold start minutes → seconds; directly answers the 100GB+ problem |
All additive to today's stack — Router preview, RoCE fabric, and the optimized image are natural landing spots. Their Router currently picks models on cost/quality; replica-level cache-awareness is the gap my PoC addresses.
Close
Phased roadmap — each phase independently valuable
1
Cache-aware routing + prefix caching
weeks, software-only · 3–7x multi-turn TTFT (proven: PoC + Google prod)
2
FP8 defaults + speculative decoding
one quarter · 2–3x TPOT, lossless via verification
3
KV pooling + P/D disaggregation
2–3 quarters, needs RDMA · tail SLOs at high utilization
4
NVFP4 + sparse attention
rides B300 ramp · next 2–4x cost curve
The through-line: every phase attacks bytes-per-token or bytes-not-recomputed; every phase is measured by the same four KPIs (TTFT, TPOT, ITL, tokens/sec/dollar) and gated by the same quality bar. Nothing is speculative — each item ships today in llm-d, vLLM, SGLang, or DeepSeek's open stack.
Appendix (→): tech choices & rejected alternatives · five failure-mode walkthroughs · throughput math & 10x plan · references.
End here for the main talk; everything after is Q&A ammunition.
Appendix A1 — demo repo
Tech choices & the alternatives we rejected
| Choice | What we use | Rejected alternative | Why |
|---|
| Inference engine | llama.cpp (llama-server, Metal) | vLLM · Ollama | vLLM has no Mac GPU path; Ollama hides needed flags (cache_prompt, slots). Fleet answer: vLLM/SGLang/TensorRT-LLM. |
| Framework | Python stdlib (urllib, subprocess) | FastAPI + aiohttp + locust | Zero dependencies = runs anywhere in the room; the harness is the product. |
| Storage | Flat results.json / md | SQLite / Postgres | KBs of data, git-diffable; a DB adds ops surface for zero queries. |
| Queue | llama-server in-process slot queue (1 slot) | Redis / Celery | Benchmark is deliberately sequential — an external queue adds latency noise. Production needs real admission control (A3). |
| Deploy target | Local Apple Silicon | Docker on GPU Droplet | Must run offline mid-interview; methodology transfers to DOKS + GPU Droplets unchanged. |
Every choice optimizes reproducibility-in-the-room over production fidelity — deliberately, with a named production substitute for each.
Appendix A2 — failure modes 1–3 (bench.py)
Crash, restart, duplicate
1 · Worker (llama-server) crashes mid-job
Current: stream read raises → finally: (bench.py:205) stops the server — but results are built in one list (:213) and written once at the end (:215). Risk: crash at the last quant loses the whole run. Fix: checkpoint — json.dump after each bench_quant().
2 · Process restarts — what state is lost?
Current: all results in memory until the final write — total loss. Worse: the child server is orphaned holding port 8933; the next run's health poll (:69) gets 200 OK from the old server and silently benchmarks the wrong model. Risk: silent wrong-model data. Fix: atexit kill + verify served model identity via /props before measuring.
3 · Duplicate submission
Current: harness is sequential, so no dupes — but a duplicated prompt would hit the slot cache and record a warm prefill as cold; that's why every cold measurement carries a nonce (:159, :186). Server-side there's no idempotency: a retried POST generates (and would bill) twice. Fix: idempotency keys at the gateway; keep nonce discipline in the harness.
The orphan-server case is the sneaky one — the health check passes for the wrong reason. Identity-check what you benchmark.
Appendix A3 — failure modes 4–5 + backpressure
Burst traffic and the job that never finishes
4 · Queue full under burst
Current: server launched without --parallel (bench.py:57–64) → one slot. Concurrent requests queue inside llama-server; each client waits up to the 600s socket timeout (:109). No 429, no shedding. Risk: head-of-line blocking → unbounded latency → timeout avalanche, zero signal to callers. Fix: --parallel N + gateway admission control returning 429 + Retry-After past a queue-depth threshold.
5 · A job that never finishes
Current: two hazards. (a) Health poll is while True (:69), no deadline — an alive-but-never-healthy server (bad model file, VRAM exhaustion) loops forever. (b) The 600s timeout is per-socket-read — a stream trickling one token per 599s never times out; only max_tokens bounds it. Fix: health-poll deadline (~120s) + wall-clock budget per request in the client.
Production translation: these five map to checkpointed state, identity verification, idempotency keys, admission control, and request deadlines — the standard reliability kit, discovered bottom-up from 220 lines of Python.
The point isn't that a demo script has gaps — it's knowing precisely where they are and what each costs.
Appendix A4 — capacity
Throughput math from actual config, and the 10x path
Today (measured constants · 1 slot · ctx 8192)
- Max concurrent jobs: 1. Everything else queues (no
--parallel).
- 256-token response ≈ prefill 0.05–0.45s + decode 256/176.5 ≈ 1.45s (Q4) → ~1.5–1.9s/request → ~0.5–0.7 req/s; FP16 ≈ 0.3 req/s.
- Health-poll sleep 0.1s (:78) ≤ 100ms per server start — noise.
- First bottleneck: single-slot serialization — not CPU or RAM. At batch 1, 1.1–3.6GB of weights stream per token for one stream; bandwidth amortizes across zero neighbors.
Scaling to 10x — what changes first
- First:
--parallel 8 + continuous batching — decode steps share each weight pass; ~5–8x for mild ITL cost. KV check: 8 slots x 8192 ctx x ~110MB ≈ 0.9GB — fits.
- Then: checkpointing + admission control (A2/A3) so the load is survivable, not just fast.
- Then: replicas behind the cache-aware router — the PoC already proves the routing layer; linear scale-out with preserved hit rates.
- Bottleneck order: slots → KV memory → memory bandwidth → compute. Same order holds on the fleet.
Rough math from real config values — and the generalization: batching first, memory second, silicon last.
Appendix A5
Selected references
Systems canon
- Orca (OSDI '22) — continuous batching
- vLLM / PagedAttention (SOSP '23) — arXiv:2309.06180
- DistServe (OSDI '24) — P/D disaggregation, goodput — arXiv:2401.09670
- Sarathi-Serve (OSDI '24) — chunked prefill — arXiv:2403.02310
- Mooncake (FAST '25 Best Paper) — KV-centric serving — arXiv:2407.00079
- FlashAttention 1–4 — arXiv:2205.14135 · together.ai/blog/flashattention-4
2026 techniques used in this deck
- llm-d (CNCF) — cache-aware routing — github.com/llm-d
- GKE Inference Gateway — 35% TTFT / 2x p95 in prod — Google Cloud blog
- DeepSeek open-infra — FlashMLA, DeepEP, EPLB · 73.7k/14.8k tok/s/node
- NVFP4 — NVIDIA dev blog · ~99% recovery at 70B+ (Red Hat evals)
- EAGLE-3 — arXiv:2503.01840 · DSA sparse attention — arXiv:2512.02556
- LMCache + Mooncake Store — vLLM cluster KV pooling (May 2026)
Full annotated bibliography: inference-trends-2026.md §6 · Raw data: demo/results/results.json, routing.json · Written spec: spec.html
← → to navigate · Home/End to jump