Before generation
Prehoc
The router picks which model to call before any tokens are produced. This is the practical deployment setting: one model per request, chosen up front.
Interactive artifacts and guides to contextualize our findings
Detailed
Benchmarks
Benchmarks
See an interactive benchmark level breakdown of individual LLM performance and the AI Frontier
Select a benchmark to view the chart
Choose any benchmark to see the results in the graph
Average over
all 16 benchmarks
all 16 benchmarks
Unweighted average of all 16 benchmarks. Quality is an arithmetic mean; cost is a geometric mean (the exponential of the mean log cost) so expensive agentic tasks do not dominate cheap Q&A.
How to:
read this data
How the Capability Frontier is built from 10 runs per model, what prehoc and posthoc points mean, and a walkthrough of the construction.
Every LLM is sampled 10 times on each datapoint. Those ten independent generations are the raw material for both frontiers below: a prehoc oracle (choose a model before seeing an answer) and a posthoc curve (generate, then keep the best).
After generation
Posthoc
Several answers are generated, then a judge keeps the best. Stronger than prehoc, but the plotted bound assumes a perfect, free verifier.
Prehoc oracle
Point estimates on the curve
For each request the oracle scores every model with a weighted mix of quality and cost, then picks the best. Sweeping the weight from cheapest to highest quality traces the prehoc Capability Frontier.
φ(α) = α · Q* + (1 − α) · (−C*)
Q* and C* are min–max normalized quality and cost. α = 1 is pure quality; α = 0 is pure cost. Each α is one point on the prehoc curve.
Taking the best sample mean across models overstates what is achievable: lucky draws get selected. With only 10 generations, that bias is material. We fit how the naive oracle changes with the number of generations G and take the large-G intercept as the debiased estimate.
O(G) = a + b · G⁻λ
a is the debiased oracle — the quality or cost you would expect with infinitely many samples. λ is the observed decay of the finite-sample bias. The plotted prehoc points use this extrapolated intercept, not the raw 10-run maximum.
Posthoc curve
Points 1 through 10
After answers exist, a judge can keep the best of k independent samples. The posthoc curve marks that best-of-k result for k = 1 … 10, matching the ten generations we collected per model per datapoint.
Point 1
A single draw. No second chance — the same information as one prehoc call, before a judge re-ranks.
Points 2–9
Best of k samples. Quality rises as the judge has more independent tries to choose from, at higher total generation cost.
Point 10
Best of all ten collected generations. This is the most naive posthoc bound in the paper: every model is queried on every prompt, then the best sample is kept.
Agentic tasks
A lower bound, not the full oracle
Agentic benchmarks (Terminal-Bench, agentic LiveCodeBench) unfold over many steps. The best model can change mid-trajectory — a cheap model for a listing step, a stronger one for a patch. The true oracle is combinatorially hard: it would pick a model at every step.
We instead fix one LLM for the whole trajectory. Routing can still choose which model runs that trajectory, but it cannot switch models between steps. That understates what per-step routing could achieve, so the agentic frontier is a lower bound on the true Capability Frontier.
Assumptions and full derivations are in the paper — including the smooth-transition fit used when G is small, why the posthoc bound is an upper envelope rather than a deployable router, and this agentic single-trajectory simplification. Read the paper.
Watch the construction
Coming
soon
soon
What lands next: the LLMs queued for the following benchmark run, and the analyses we are building on top of the Capability Frontier.
12 new LLMs
Models joining the frontier
These LLMs are queued for the next benchmark run. Each one is sampled ten times on every datapoint before it can appear on a curve, so they are absent from the charts above.
- Gpt 5.6 Sol
- Gpt 5.6 Terra
- Gpt 5.6 Luna
- Claude Fable 5
- Claude Opus 5
- Claude Sonnet 5
- Gemini 3.7 Flash
- Muse Spark 1.2
- Kimi K3
- Minimax M2.7
- Grok 4.6
- Qwen3.8 Max
Third axis
Latency analysis
Quality and cost are only two of the three numbers teams actually budget for. We are measuring time-to-first-token and end-to-end latency per model so the frontier can be read as a three-way tradeoff rather than a flat curve.
Composition
LLM mixture percentages
Every point on the multi-model frontier is a blend of LLMs, not a single winner. We will surface the exact mixture behind each point — which models the router calls, and what share of requests each one takes.
Marginal value
How individual LLMs move the frontier
Drop one LLM from the pool and rebuild the curve to isolate what that model contributes. This separates the models that genuinely extend the frontier from the ones another model already covers.