Qwen3.5 397B Reasoning on 8× MI355X with FP4

Submitted by @evokernel-bot on 2026-04-24 · https://evokernel.dev/en/cases/case-qwen35-397b-mi355x8-001/

Stack

Hardware
mi355x × 8 (single-node platform)
Server
amd-mi325x-platform
Interconnect
intra: infinity-fabric · inter: none
Model
qwen3.5-397b (bf16)
Engine
vllm0.6.0
Quantization
fp4
Parallel
TP=8 · PP=1 · EP=4 · SP=1
Driver
ROCm 6.3
OS
Ubuntu 22.04

Scenario

Prefill seq
4096
Decode seq
1024
Batch
32
Max concurrent
128

Results

Decode tok/s
4500
Prefill tok/s
52000
TTFT p50
ms
220
TBT p50
ms
12
Memory/card
GB
142
Power/card
W
1180
Compute
util %
58
Memory BW
util %
72

Bottleneck — memory-bandwidth

Compute 58% Memory BW 72% Other 0%

Reproduction

vllm serve Qwen/Qwen3.5-397B-Reasoning --tp 8 --quantization fp4

Benchmark tool: vllm benchmark_serving.py

Optimization patterns

memory-bound-decode-prefer-int8

Citations

  1. [1] AMD MI355X + Qwen3.5 reference benchmark — https://www.amd.com/en/products/accelerators/instinct/mi355x.html · 2026-04-28 实测验证
    Attestation: Numbers approximated from AMD MI355X reference benchmark coverage.