G Groq Last verified

Groq LPU (TSP v1)

PCIE 在售 发布于 2024 groq-tsp-gen1
BF16
TFLOP/s
188 厂商声称
FP8
TFLOP/s
750 厂商声称
FP4
TFLOP/s
不支持
Memory
GB
0.23 厂商声称
Mem BW
GB/s
80000 厂商声称
TDP
W
215 厂商声称

完整规格

算力

FP4 TFLOPS
不支持
FP8 TFLOPS
750
BF16 TFLOPS
188
FP16 TFLOPS
188
INT8 TOPS
750

显存

容量
0.23 GB
带宽
80000 GB/s
类型
on-die-sram

芯片架构 🟢 vendor floorplan

PEs count
5120
制程
14 nm
PCIe
Gen 4 ×16

Scale-Up (节点内)

协议
GroqRack-Mesh
单链带宽
160 GB/s
World size
8
拓扑
switched
交换机
—

Scale-Out (节点间)

单卡出口
80 Gbps
协议
Ethernet-RoCE
NIC
—

拓扑示意

拓扑结构 · Topology
8 卡 scale-up domain
芯片内部 / Die-level architecture
on-die-sram 0.23 GB @ 80.0 TB/s Groq LPU (TSP v1) L2 / shared cache · NoC L1$ / register file (per PEs) 5120 PEss · darker block = tensor / matrix engine 188 TFLOPS BF16 · 750 FP8 · 0.23 GB on-die-sram @ 80.0 TB/s · 215 W TDP

🟢 vendor floorplan 5,120 PEss · 230 MB on-die SRAM · 14 nm

📦 on-die SRAM only (capacity ≪ HBM, bandwidth ≫ HBM) ⏱ deterministic latency (p99 ≈ p50)
集群拓扑 / Cluster topology · GroqRack-Mesh @ 160 GB/s
GroqRack-Mesh switch 160 GB/s/link · all-to-all GPU 0 0.23GB GPU 1 0.23GB GPU 2 0.23GB GPU 3 0.23GB GPU 4 0.23GB GPU 5 0.23GB GPU 6 0.23GB GPU 7 0.23GB 8 cards · switched topology · scale-out: 80 Gbps/card
Scale-Up · 域内
GroqRack-Mesh
160 GB/s · 拓扑: switched
world_size = 8
Scale-Out · 跨域
Ethernet-RoCE
80 Gbps/卡 NIC

能跑哪些模型?

Quick estimates · decode tok/s/card 上界

TP=8 · FP8 · batch=16 · prefill=1024 · decode=256 · 已应用 efficiency 校准

在计算器中调整 →
模型 参数 (active) Decode tok/s/card 瓶颈
DeepSeek V4 Pro
deepseek
 49B — 显存不足
DeepSeek V4 Flash
deepseek
 13B — 显存不足
Mistral Small 4
mistral
 22B — 显存不足
GLM-5 Reasoning
zhipu
 32B — 显存不足
GLM-5.1
zhipu
 32B — 显存不足
Qwen3.6 Plus
alibaba
 35B — 显存不足
Kimi K2.6
moonshot
 32B — 显存不足
MiniMax M2.7
minimax
 46B — 显存不足

算子级 fit · 任意模型瓶颈类型 + 上界

算子级 fit · operator-level fit (per-token roofline)

基于每个模型 operator_decomposition + 本卡 BF16 188 TFLOPS / 80,000 GB/s 计算 · ridge point ≈ 2 FLOPs/byte

上界 = min(计算屋顶, 内存带宽屋顶) · efficiency 未应用
模型 domain 主导算子 AI · F/B 瓶颈 tok/s 上界
GraphCast scientific graph-message-passing 0.9 💾 内存带宽 148k
AlphaFold 3 scientific pair-bias-attention 2.3 💾 内存带宽 44k
DeepSeek V4 Pro llm matmul 245.5 🔥 计算 31k
GPT-OSS llm matmul 0.7 💾 内存带宽 6468
Gemma 4 26B llm matmul 0.7 💾 内存带宽 4807
DeepSeek V4 Flash llm matmul 0.8 💾 内存带宽 4550
Mistral Small 4 llm matmul 0.6 💾 内存带宽 2073
Llama 4 Maverick llm matmul 0.8 💾 内存带宽 2048
需要 efficiency 校准 + concurrency 扫描 + TCO 估算 → 在计算器中评估 →

算子支持 & 优化空间

算子支持 & 优化空间 / Operator support & headroom

Per-operator support derived from software_support.engines + scale-up topology. Optimization headroom from measured efficiency factor.

Optimization headroom
+50 pp
moderate

No cases yet — using default 0.5 efficiency. Real headroom unknown until first measurement lands.

Communication (collective)
All-to-All 🟢 mature
all-to-all via GroqRack-Mesh world_size=8
AllReduce 🟢 mature
GroqRack-Mesh ring all-reduce
Attention
Multi-Head Attention 🟢 mature
paged-attention via vLLM/SGLang/MindIE
FlashAttention-3 🟢 mature
FA-3 on modern engine + tensor cores
Matrix multiply (GEMM)
Matrix Multiplication 🟢 mature
GEMM supported on all inference engines
MoE routing
MoE Routing 🟢 mature
MoE gating supported via vLLM ≥0.4 / SGLang
Normalization
RMSNorm 🟢 mature
fused into engine kernels
Embedding
Rotary Position Embedding 🟢 mature
fused into engine kernels
Activation
SiLU / Swish 🟢 mature
fused into engine kernels
Softmax 🟢 mature
fused into engine kernels

最接近的替代卡 (按规格相似度)

基于 BF16 算力 / 显存 / 显存带宽 / FP8 加权欧氏距离。供选型决策参考。

平头哥 含光 800
平头哥
99%
相似度
BF16
—
Mem
16
BW
0.3
⇄ 国产 替代
Tenstorrent Wormhole n300
Tenstorrent
99%
相似度
BF16
233
Mem
24
BW
0.6
AWS Inferentia 2
AWS Annapurna Labs
99%
相似度
BF16
190
Mem
32
BW
0.8
壁仞 BR104
壁仞科技
99%
相似度
BF16
128
Mem
32
BW
1.1
⇄ 国产 替代

软件栈支持

引擎 状态 BF16FP16FP4FP8 E4M3FP8 E5M2INT4 AWQ
HanGuangAI 未确认 — — — — — —
LMDeploy 未确认 — — — — — —
MindIE 未确认 — — — — — —
MoRI 未确认 — — — — — —
SGLang 未确认 — — — — — —
TensorRT-LLM (Dynamo) 未确认 — — — — — —
vLLM 社区 ✓ ✓ — ✓ — —

已有部署案例 (0)

暂无该硬件的实测案例。 成为第一个贡献者?

引证

  1. [1] Groq LPU specifications — 230 MB SRAM, 80 TB/s memory bandwidth (on-die), deterministic single-batch latency. TSP v1 first-generation at 14nm Global Foundries. — https://groq.com/lpu-inference-engine/ · 访问于 2026-04-29 厂商声称
⚠ Memory capacity is on-die SRAM only; the LPU has NO DRAM. Models must fit in 230 MB or be sharded across cards.
⚠ Roofline analysis is unusual: memory-bandwidth ceiling is dramatically higher than HBM cards but capacity is dramatically lower.
⚠ Deterministic latency makes this the only chip in the corpus where p99 ≈ p50 by design — useful for SLA-bound serving.