AWS Annapurna Labs Last verified 2026-04-29

AWS Trainium 2

Name: AWS Trainium 2
Brand: AWS Annapurna Labs
SKU: trainium-2

PROPRIETARY In production Released 2024 trn2

BF16

TFLOP/s

650 厂商声称

FP8

TFLOP/s

1300 厂商声称

FP4

TFLOP/s

unsupported

Memory

96 厂商声称

Mem BW

GB/s

2900 厂商声称

TDP

500 厂商声称

Full specs

Compute

FP4 TFLOPS

unsupported

FP8 TFLOPS

1300

BF16 TFLOPS

650

FP16 TFLOPS

650

INT8 TOPS

1300

Memory

Capacity

96 GB

Bandwidth

2900 GB/s

Type

HBM3

Die architecture 🟢 vendor floorplan

XPU count

HBM stacks

Process

5 nm

Scale-Up (intra-node)

Protocol

NeuronLink

Per-link BW

1280 GB/s

World size

Topology

ring-mesh

Switch

—

Scale-Out (inter-node)

Per-card NIC

400 Gbps

Protocol

EFA

NIC

—

Topology

拓扑结构 · Topology

64 卡 scale-up domain

芯片内部 / Die-level architecture

🟢 vendor floorplan 8 XPUs · 4× HBM · 5 nm

集群拓扑 / Cluster topology · NeuronLink @ 1280 GB/s

Scale-Up · 域内

NeuronLink

1280 GB/s · 拓扑: ring-mesh

world_size = 64

Scale-Out · 跨域

EFA

400 Gbps/卡 NIC

Which models can it run?

Quick estimates · decode tok/s/card 上界

TP=8 · FP8 · batch=16 · prefill=1024 · decode=256 · 已应用 efficiency 校准

在计算器中调整 →

模型	参数 (active)	Decode tok/s/card	瓶颈
DeepSeek V4 Pro deepseek	49B	—	显存不足
DeepSeek V4 Flash deepseek	13B	247	内存带宽
Mistral Small 4 mistral	22B	113	内存带宽
GLM-5 Reasoning zhipu	32B	93	内存带宽
GLM-5.1 zhipu	32B	—	显存不足
Qwen3.6 Plus alibaba	35B	61	内存带宽
Kimi K2.6 moonshot	32B	—	显存不足
MiniMax M2.7 minimax	46B	41	内存带宽

Operator-level fit · per-model bottleneck + upper bound

算子级 fit · operator-level fit (per-token roofline)

基于每个模型 operator_decomposition + 本卡 BF16 650 TFLOPS / 2,900 GB/s 计算 · ridge point ≈ 224 FLOPs/byte

上界 = min(计算屋顶, 内存带宽屋顶) · efficiency 未应用

模型	domain	主导算子	AI · F/B	瓶颈	tok/s 上界
DeepSeek V4 Pro	llm	matmul	245.5	🔥 计算	108k
GraphCast	scientific	graph-message-passing	0.9	💾 内存带宽	5351
AlphaFold 3	scientific	pair-bias-attention	2.3	💾 内存带宽	1608
GPT-OSS	llm	matmul	0.7	💾 内存带宽	234
Gemma 4 26B	llm	matmul	0.7	💾 内存带宽	174
DeepSeek V4 Flash	llm	matmul	0.8	💾 内存带宽	165
Mistral Small 4	llm	matmul	0.6	💾 内存带宽	75
Llama 4 Maverick	llm	matmul	0.8	💾 内存带宽	74

需要 efficiency 校准 + concurrency 扫描 + TCO 估算 → 在计算器中评估 →

Operator support & optimization headroom

算子支持 & 优化空间 / Operator support & headroom

Per-operator support derived from software_support.engines + scale-up topology. Optimization headroom from measured efficiency factor.

Optimization headroom

+-50 pp

saturated

Near saturation at 150% of roofline. Further gains require workload restructure (disaggregated, speculative, smaller batch).

Communication (collective)

All-to-All 🟢 mature

all-to-all via NeuronLink world_size=64

AllReduce 🟢 mature

NeuronLink ring all-reduce

Attention

Multi-Head Attention 🟢 mature

paged-attention via vLLM/SGLang/MindIE

FlashAttention-3 🟢 mature

FA-3 on modern engine + tensor cores

Matrix multiply (GEMM)

Matrix Multiplication 🟢 mature

GEMM supported on all inference engines

MoE routing

MoE Routing 🟢 mature

MoE gating supported via vLLM ≥0.4 / SGLang

Normalization

RMSNorm 🟢 mature

fused into engine kernels

Embedding

Rotary Position Embedding 🟢 mature

fused into engine kernels

Activation

SiLU / Swish 🟢 mature

fused into engine kernels

Softmax 🟢 mature

fused into engine kernels

最接近的替代卡 (按规格相似度)

基于 BF16 算力 / 显存 / 显存带宽 / FP8 加权欧氏距离。供选型决策参考。

BF16: 432
Mem: 96
BW: 2.5

BF16: 459
Mem: 95
BW: 2.8

BF16: 192
Mem: 96
BW: 2.4

⇄ 国产替代

NVIDIA H100 SXM5 80GB

NVIDIA

99%

相似度

BF16: 989
Mem: 80
BW: 3.4

Software-stack support

Engine	Status	BF16	FP16	FP4	FP8 E4M3	FP8 E5M2	INT4 AWQ
HanGuangAI	unconfirmed	—	—	—	—	—	—
LMDeploy	unconfirmed	—	—	—	—	—	—
MindIE	unconfirmed	—	—	—	—	—	—
MoRI	unconfirmed	—	—	—	—	—	—
SGLang	unconfirmed	—	—	—	—	—	—
TensorRT-LLM (Dynamo)	unconfirmed	—	—	—	—	—	—
vLLM	community	✓	✓	—	✓	—	—

Measured efficiency factor

Computed from 1 measured cases for this card. The calculator uses this value in place of the default 0.5.

1.50

measured / theoretical (n=1)

Existing deployment cases (1)

DeepSeek V3 on AWS Trainium 2 (64-chip Trn2 instance)

deepseek-r1 · vllm 0.6.0 · bf16 · TP=16

3600 tok/s

2026-04-19

Citations

[1] AWS Trainium 2 product page — https://aws.amazon.com/ai/machine-learning/trainium/ · accessed 2026-04-28 厂商声称
[2] Trainium 2: 8 NeuronCore-v3 engines, 4× HBM3 ⇒ 96 GB; NeuronLink-v3 fabric scales to 64 chips per Trn2 UltraServer; TSMC 5nm-class — https://awsdocs-neuron.readthedocs-hosted.com/en/latest/general/arch/neuron-hardware/trainium2.html · accessed 2026-04-28 厂商声称

⚠ AWS does not sell chips; only available via EC2 Trn2 instances.