By Mona Mishra · Extension of the Roofline Model for MoE Workloads & Inference Cost Economics
The key insight from recent research: The standard Roofline Model assumes dense, sequential token generation. This tool extends it in ways that matter for real serving decisions — (1) Mixture-of-Experts (MoE) activates only a fraction of parameters per token, reshaping the memory access pattern and the model-selection tradeoff; (2) prefill and decode have arithmetic intensities that differ by 100–500×, which is the argument for disaggregated serving on specialized hardware; (3) every efficiency gain above shows up directly in cost per token, where the engineering argument becomes a budget argument.
Each dot is an open-weight model from the HuggingFace Open LLM Leaderboard (Apr 2026). X-axis = active parameters per token — the key driver of per-token memory bandwidth cost. Y-axis = MMLU score — capability proxy. MoE models reach frontier capability at a fraction of the active parameter cost of equivalent dense models.
MoE models load only active parameters per token — which nominally improves per-token memory cost. But expert routing across a batch can force loading more experts than K. This tool quantifies the "expert loading overhead" and shows where the real operating point falls on the Roofline.
Configuration
1128
1.0× (ideal)3.0× (full batch routing)
Active Params
—
Ridge Point
—
Arith. Intensity
—
MXU Utilization
—
Roofline Chart
Disaggregated Prefill / Decode — The Case for Specialized Hardware
Prefill (processing the prompt) and decode (generating tokens) have arithmetic intensities that differ by 100–500×. They are fundamentally different compute problems forced to run on the same chip. This structural mismatch means no single chip can be optimal for both. SPAD (arXiv 2510.08544) shows that specialized chips for each phase reduce hardware costs by 19–41%. This tab makes the mismatch visible.
Configuration
1128
1288192
Decode AI
—
Prefill AI
—
Decode MXU Util.
—
Prefill MXU Util.
—
Both Workloads on One Roofline
HARDWARE CO-DESIGN IMPLICATIONS
DECODE CHIP
Optimize for memory bandwidth, not peak compute. A low ridge point means the decode operating point is closer to the chip's ceiling. High HBM bandwidth, moderate TFLOPS.
PREFILL CHIP
Optimize for peak compute. Prefill's high arithmetic intensity means the MXU can sustain near-peak utilization. More MXU cores, higher TFLOPS, lower per-byte HBM cost acceptable.
TCO IMPACT
SPAD (2025): specialized chips per phase reduce hardware cost by 19–41% for the same throughput — by eliminating the silicon compromise of a generalist design.
SYNTHESIS — WHAT THIS MEANS FOR HARDWARE DESIGN
① The binding constraint is memory, not compute
Dense decode at batch=1 runs at ~0.2% MXU utilization. Adding more TFLOPS does not help. The hardware roadmap question is: how do we move the workload rightward on the Roofline — through batching, quantization, or architecture changes — not: how do we add more raw compute?
② MoE is an architectural response to a hardware constraint
MoE reduces active parameters per token, shrinking the bytes-per-token load — the direct lever the model-selection tab above is built around. It is a model-architecture strategy that raises arithmetic intensity, making the workload a better fit for existing silicon without new hardware.
③ The next hardware frontier is co-design, not scale
Disaggregated prefill/decode chips, MoE-aware routing silicon, and speculative-decoding-aware memory subsystems are all examples of hardware shaped by workload structure. The 19–41% TCO reduction from SPAD-style specialization is not a software optimization — it is recovered by designing the chip around what the workload actually needs.
By Mona Mishra · Based on MoE-SpeQ, SPAD, and HuggingFace Open LLM Leaderboard v2
Effective Cost per 1M Tokens
Throughput × utilization × cloud list price. Cloud pricing was verified against live provider pages on 2026-07-12; throughput is an illustrative, not current-gen, benchmark figure — see the note below the raw hardware comparison.
Configuration
10%100%
Cost per 1M Tokens
—
Effective Throughput
—
Cost Comparison — Same Cloud, Same Utilization
Performance per Dollar — the raw hardware baseline
Peak FP16 TFLOP/s per $1,000 of list price, across chips with a public list price. This is the sticker-price ceiling, before utilization or cloud rental pricing enter the picture — it explains why a chip can look cheap here and still cost more per token above. Chips sold only through cloud rental (TPUs, Trainium) have no public list price and are excluded.
Hardware specs adapted from Epoch AI (CC-BY 4.0). Throughput figures above are illustrative, not current-gen — kept as the same older chip set as the rest of this page; see MLPerf Inference v6.0 for current results.