N NVIDIA Last verified

NVIDIA L40S

PCIE In production Released 2023 ada-lovelace
BF16
TFLOP/s
366 厂商声称
FP8
TFLOP/s
1466 厂商声称
FP4
TFLOP/s
unsupported
Memory
GB
48 厂商声称
Mem BW
GB/s
864 厂商声称
TDP
W
350 厂商声称

Full specs

Compute

FP4 TFLOPS
unsupported
FP8 TFLOPS
1466
BF16 TFLOPS
366
FP16 TFLOPS
366
INT8 TOPS
1466

Memory

Capacity
48 GB
Bandwidth
864 GB/s
Type
GDDR6

Die architecture 🟢 vendor floorplan

SM count
142
Tensor cores / SM
4
L2 cache
96 MB
Process
5 nm
Die area
609 mm²
Transistors
76 B
PCIe
Gen 4 ×16

Scale-Up (intra-node)

Protocol
PCIe-Gen4
Per-link BW
64 GB/s
World size
8
Topology
pcie-fabric
Switch
—

Scale-Out (inter-node)

Per-card NIC
200 Gbps
Protocol
InfiniBand-NDR
NIC
ConnectX-7

Topology

拓扑结构 · Topology
8 卡 scale-up domain
芯片内部 / Die-level architecture
GDDR6 48 GB @ 0.9 TB/s NVIDIA L40S L2 / shared cache · NoC L1$ / register file (per SM) 142 SMs · darker block = tensor / matrix engine 366 TFLOPS BF16 · 1466 FP8 · 48 GB GDDR6 @ 0.9 TB/s · 350 W TDP

⚠ illustrative / 示意性版图: compute-unit and HBM-stack count are inferred from public BF16 / memory specs. architecture field not populated for this card yet. Contribute floorplan data →

集群拓扑 / Cluster topology · PCIe-Gen4 @ 64 GB/s
PCIe-Gen4 switch 64 GB/s/link · all-to-all GPU 0 48GB GPU 1 48GB GPU 2 48GB GPU 3 48GB GPU 4 48GB GPU 5 48GB GPU 6 48GB GPU 7 48GB 8 cards · pcie-fabric topology · scale-out: 200 Gbps/card
Scale-Up · 域内
PCIe-Gen4
64 GB/s · 拓扑: pcie-fabric
world_size = 8
Scale-Out · 跨域
InfiniBand-NDR
200 Gbps/卡 NIC
ConnectX-7

Which models can it run?

Quick estimates · decode tok/s/card 上界

TP=8 · FP8 · batch=16 · prefill=1024 · decode=256 · 已应用 efficiency 校准

在计算器中调整 →
模型 参数 (active) Decode tok/s/card 瓶颈
DeepSeek V4 Pro
deepseek
 49B — 显存不足
DeepSeek V4 Flash
deepseek
 13B 74 内存带宽
Mistral Small 4
mistral
 22B 34 内存带宽
GLM-5 Reasoning
zhipu
 32B 28 内存带宽
GLM-5.1
zhipu
 32B — 显存不足
Qwen3.6 Plus
alibaba
 35B — 显存不足
Kimi K2.6
moonshot
 32B — 显存不足
MiniMax M2.7
minimax
 46B — 显存不足

Operator-level fit · per-model bottleneck + upper bound

算子级 fit · operator-level fit (per-token roofline)

基于每个模型 operator_decomposition + 本卡 BF16 366 TFLOPS / 864 GB/s 计算 · ridge point ≈ 424 FLOPs/byte

上界 = min(计算屋顶, 内存带宽屋顶) · efficiency 未应用
模型 domain 主导算子 AI · F/B 瓶颈 tok/s 上界
DeepSeek V4 Pro llm matmul 245.5 💾 内存带宽 35k
GraphCast scientific graph-message-passing 0.9 💾 内存带宽 1594
AlphaFold 3 scientific pair-bias-attention 2.3 💾 内存带宽 479
GPT-OSS llm matmul 0.7 💾 内存带宽 70
Gemma 4 26B llm matmul 0.7 💾 内存带宽 52
DeepSeek V4 Flash llm matmul 0.8 💾 内存带宽 49
Mistral Small 4 llm matmul 0.6 💾 内存带宽 22
Llama 4 Maverick llm matmul 0.8 💾 内存带宽 22
需要 efficiency 校准 + concurrency 扫描 + TCO 估算 → 在计算器中评估 →

Operator support & optimization headroom

算子支持 & 优化空间 / Operator support & headroom

Per-operator support derived from software_support.engines + scale-up topology. Optimization headroom from measured efficiency factor.

Optimization headroom
+-50 pp
saturated

Near saturation at 150% of roofline. Further gains require workload restructure (disaggregated, speculative, smaller batch).

Communication (collective)
All-to-All 🟢 mature
all-to-all via PCIe-Gen4 world_size=8
AllReduce 🟢 mature
PCIe-Gen4 ring all-reduce
Attention
Multi-Head Attention 🟢 mature
paged-attention via vLLM/SGLang/MindIE
FlashAttention-3 🟢 mature
FA-3 on modern engine + tensor cores
Matrix multiply (GEMM)
Matrix Multiplication 🟢 mature
GEMM supported on all inference engines
MoE routing
MoE Routing 🟢 mature
MoE gating supported via vLLM ≥0.4 / SGLang
Normalization
RMSNorm 🟢 mature
fused into engine kernels
Embedding
Rotary Position Embedding 🟢 mature
fused into engine kernels
Activation
SiLU / Swish 🟢 mature
fused into engine kernels
Softmax 🟢 mature
fused into engine kernels

最接近的替代卡 (按规格相似度)

基于 BF16 算力 / 显存 / 显存带宽 / FP8 加权欧氏距离。供选型决策参考。

摩尔线程 MTT S4000
摩尔线程
99%
相似度
BF16
100
Mem
48
BW
0.8
⇄ 国产 替代
寒武纪 MLU370-X8
寒武纪
99%
相似度
BF16
96
Mem
48
BW
0.6
⇄ 国产 替代
昇腾 910B
华为昇腾
99%
相似度
BF16
320
Mem
64
BW
1.6
⇄ 国产 替代
寒武纪 思元 590
寒武纪
99%
相似度
BF16
256
Mem
64
BW
1.2
⇄ 国产 替代

Software-stack support

Engine Status BF16FP16FP4FP8 E4M3FP8 E5M2INT4 AWQ
HanGuangAI unconfirmed — — — — — —
LMDeploy official ✓ ✓ — ✓ ✓ ✓
MindIE unconfirmed — — — — — —
MoRI unconfirmed — — — — — —
SGLang official ✓ ✓ — ✓ ✓ ✓
TensorRT-LLM (Dynamo) official ✓ ✓ — ✓ ✓ ✓
vLLM official ✓ ✓ — ✓ ✓ ✓
Measured efficiency factor

Computed from 1 measured cases for this card. The calculator uses this value in place of the default 0.5.

1.50
measured / theoretical (n=1)

Existing deployment cases (1)

Citations

  1. [1] NVIDIA L40S Datasheet (Ada Lovelace inference-optimized PCIe card) — https://www.nvidia.com/en-us/data-center/l40s/ · accessed 2026-04-28 厂商声称
  2. [2] NVIDIA Ada Lovelace Whitepaper (AD102 die: 142 SMs enabled in L40S, 96 MB L2, 76B transistors, 609 mm² @ TSMC 4N) — https://images.nvidia.com/aem-dam/Solutions/Data-Center/l4/nvidia-ada-gpu-architecture-whitepaper-v2.1.pdf · accessed 2026-04-28 厂商声称
⚠ L40S uses GDDR6 (not HBM); memory bandwidth is much lower than H100/A100, making it bandwidth-bound for many decode workloads.
⚠ Designed for graphics + inference workloads (gaming derivative); strong FP8/INT8 throughput per dollar.
⚠ All performance figures are vendor-claimed unless tier=measured.