Frontier Language Models Under a Verifiable Protocol: Accuracy, Cost and Speed on Expert-Level Questions

This paper reports results for 31 frontier language models under the methodology described in the companion paper, "How Every Number Is Produced: A Verifiable Methodology for Evaluating Large Language Models". A model is classed as frontier when its blended list price, (3 x input + 1 x output) / 4 in USD per million tokens from the OpenRouter catalogue at test time, is at least $1.00; every other model is non-frontier. The companion results paper covers the 41 non-frontier models evaluated under the identical methodology.

I report what the measurements support and no more: a score with its 95% interval for every model, which differences from the leading model pass a significance test, what each model cost per question as billed, and how fast it streamed. I do not combine these into a single index.

Each model answered the same 80 ten-option multiple-choice questions (50 from SuperGPQA, hard subset; 30 from MMLU-Pro) through OpenRouter at temperature 0, reasoning effort "low" where supported and at most 4000 output tokens. Answers were graded by a deterministic extraction rule against the datasets' answer keys. Models were tested between October 25, 2026 and October 27, 2026. Chance accuracy is 10%.

No request in this group failed after retries. moonshotai/kimi-k2.7-code (1), qwen/qwen3.8-2.4t-a95b (1), cohere/command-a (1), mistralai/mistral-medium-3-5 (1) and nousresearch/hermes-4-405b (1) had answers excluded because the run manifest did not confirm their proof hash (methodology, Sections 7 and 9). Scores below are therefore out of the number of graded answers shown.

Scores range from 26.6% to 85.0%, with a median of 75.0%. The highest score is anthropic/claude-opus-5.5 with 68 of 80 (85.0%; 95% CI 75.6% to 91.2%). For comparison, the median of the non-frontier group is 48.8%.

Table 2 tests the leader, anthropic/claude-opus-5.5, against every other model in the group with a two-proportion z-test on the two scores (pooled standard error). A gap counts only when |z| is at least 1.96, the usual threshold for 95% confidence on a two-sided test. Of the 30 comparisons, 11 gaps are significant and 19 are too close to call. The models too close to call against the leader are google/gemini-4-argon, google/gemini-3.8-flash, openai/gpt-5.3-codex, anthropic/claude-fable-5.1, x-ai/grok-4.7, anthropic/claude-sonnet-5.5, meta/muse-wonder, google/gemini-3.1-pro-preview, meta/muse-spark-open, moonshotai/kimi-k3.1-pro, openai/gpt-6-astra, openai/gpt-6.1-sol, openai/gpt-6.1-terra-pro, openai/gpt-5.6-terra, qwen/qwen3.8-max-0902, openai/gpt-6-sol, openai/gpt-6.1-terra, openai/gpt-5.4 and openai/o3. On this suite their order relative to the leader is unresolved, not a ranking.

As a paired check, an exact McNemar test with Holm-Bonferroni correction (last two columns) agrees with the z-test verdict in 29 of 30 comparisons. It does not confirm the z-test's verdict for deepseek/deepseek-v4-pro-0813, so that gap should be read with extra caution. The terms are explained below and in the methodology paper.

Two-proportion z-test (opens Wikipedia in a new tab)

Tests whether two success rates differ by more than chance would explain. The gap between the two scores is divided by its standard error (pooled from both models); |z| >= 1.96 corresponds to 95% confidence, two-sided.

McNemar's test (opens Wikipedia in a new tab)

A test for paired yes/no outcomes. Because both models answered the same questions, it looks only at the questions where exactly one of them was right and asks whether that split could be a coin flip. It ignores questions both got right or both got wrong.

Holm-Bonferroni correction (opens Wikipedia in a new tab)

When one model is compared with many others, some comparisons look significant by luck alone. Holm's method raises the bar step by step so that the chance of any false "significant" result across the whole set stays at 5%.

Categories hold between 6 and 30 questions each, so per-category results are shown as counts and are noisy; they indicate where models differ, not reliable sub-rankings.

Billed cost per question ranges from $0.0012 (thinkingmachines/inkling) to $0.025, with a median of $0.0088. Spearman's rank correlation between score and cost per question is 0.06: within this group, paying more per question is only weakly related to a higher score.

6 models are on the cost-accuracy Pareto set, from cheapest to most accurate: thinkingmachines/inkling (66.2%, $0.0012), openai/gpt-6.1-terra (73.8%, $0.0013), openai/gpt-6.1-sol (76.2%, $0.0029), google/gemini-3.8-flash (82.5%, $0.0037), google/gemini-4-argon (83.8%, $0.0062) and anthropic/claude-opus-5.5 (85.0%, $0.015). Every other model in the group is matched or beaten on both score and cost by at least one of these. Because scores carry the uncertainty shown in Table 1, models just off the set may be equivalent in practice.

Under the 10% rule, thinkingmachines/inkling and openai/gpt-6.1-terra cost about the same per question as the cheapest model.

Spearman's rank correlation (opens Wikipedia in a new tab)

Measures how consistently one quantity rises with another using only their rank order, from -1 to +1. Here it asks whether more expensive models tend to score higher.

Pareto set (cost-accuracy frontier) (opens Wikipedia in a new tab)

The models for which no other model is both at least as accurate and at least as cheap. Any model off the set is beaten on both counts by one on it.

Median output speed across the group is 94 tokens per second and median time to first token is 1.29 s. The highest median output speed is qwen/qwen3.5-397b-a17b at 240 tokens per second; the lowest median time to first token is nousresearch/hermes-4-405b at 0.35 s. Output speed is withheld for openai/gpt-6.1-terra-pro, whose runs delivered the answer in a single burst after a silent interval (methodology, Section 5); latency is still reported. Under the 10% rule, meta/muse-spark-open is similar in output speed to qwen/qwen3.5-397b-a17b, and moonshotai/kimi-k2.7-code is similar to nousresearch/hermes-4-405b in time to first token. Speed was measured from one client machine; absolute latencies depend on network distance to the host.

9 models returned no extractable answer on at least 8 of the graded questions: qwen/qwen3.5-397b-a17b (50), mistralai/mistral-medium-3-5 (47), mistralai/mistral-large-4-0 (35), qwen/qwen3.8-2.4t-a95b (26), moonshotai/kimi-k2.7-code (19), nvidia/nemotron-3-ultra-550b-a55b (18), stepfun/step-5-preview (17), deepseek/deepseek-v4-pro-0813 (12) and qwen/qwen3.8-max-0902 (10). Across all models in both groups, 927 of the 939 missing answers ended with finish reason "length" and 932 had an empty answer text, which points to the 4000-token limit being spent on reasoning rather than to wrong answers. These models' scores measure performance under this budget; with a larger budget they may score higher. I do not adjust for this.

The main result is a limit on what can be claimed. With 80 questions, 19 models in this group are too close to call against the leader, so a single ordered leaderboard would overstate what was measured. The data support coarser statements: which models sit clearly below the leader (Table 2), which models give the best accuracy for their billed cost (the Pareto set), and where low scores come from missing answers under a fixed token budget rather than from wrong answers.

Readers choosing a model for production work should weigh these results against task fit: the suite measures expert-level multiple-choice knowledge and reasoning, not coding, tool use or long-context work.

• The group boundary is a list-price rule; it does not imply that every model above it is more capable than every model below it.
• 80 questions in one task format; intervals are roughly +/-10 points.
• One reasoning setting and a 4000-token limit for all models.
• OpenRouter routing may serve a model from different hosts; prices, hosts and model versions change after the test dates.
• Speed and latency come from a single client machine.

Per-model results (llm-evaluation-models.csv) and per-question evidence with proof hashes and generation IDs (llm-evaluation-evidence.csv, llm-evaluation-evidence.json) are published at suhasbhairav.com/research under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Every table in this paper can be recomputed from those files.