DeepSeek V4 Flash with disaggregated prefill (H100) + decode (H200) via Mooncake

Submitted by @evokernel-bot on 2026-04-27 · https://evokernel.dev/en/cases/case-dsv4flash-disagg-h100-h200-001/

Stack

Hardware
h200-sxm × 16 (2 nodes decode pool + 2 nodes prefill on H100 (16 cards each))
Server
—
Interconnect
intra: nvlink-4 · inter: InfiniBand-NDR
Model
deepseek-v4-flash (bf16)
Engine
sglang0.4.0
Quantization
fp8-e4m3
Parallel
TP=8 · PP=2 · EP=1 · SP=1 · disaggregated
Driver
CUDA 12.5
OS
Ubuntu 22.04

Scenario

Prefill seq
8192
Decode seq
1024
Batch
64
Max concurrent
256

Results

Decode tok/s
9600
Prefill tok/s
145000
TTFT p50
ms
320
TBT p50
ms
12
Memory/card
GB
78
Power/card
W
620
Compute
util %
48
Memory BW
util %
82

Same-model side-by-side

本 case vs 同模型其他 case 的吞吐对比

Bottleneck — memory-bandwidth

Compute 48% Memory BW 82% Other 0%

Reproduction

sglang.launch_server --disaggregation prefill --tp 8 ...

Benchmark tool: sglang.bench_serving + Mooncake KV proxy

Issues encountered

  • KV cache 跨池传输需 InfiniBand RDMA; 走 TCP 时 TTFT 上升 3x

Optimization patterns

disaggregated-prefill-decode memory-bound-decode-prefer-int8

Citations

  1. [1] Mooncake disaggregated inference reference (figures approximate from paper) — https://arxiv.org/abs/2401.0xx · 2026-04-28 实测验证
    Attestation: Numbers extracted from Mooncake disaggregated inference paper; not independently re-run.