Non-Frontier Language Models Under a Verifiable Protocol: Accuracy, Cost and Speed on Expert-Level Questions

This paper reports results for 41 non-frontier language models under the methodology described in the companion paper, "How Every Number Is Produced: A Verifiable Methodology for Evaluating Large Language Models". A model is classed as frontier when its blended list price, (3 x input + 1 x output) / 4 in USD per million tokens from the OpenRouter catalogue at test time, is at least $1.00; every other model is non-frontier. The companion results paper covers the 31 frontier models evaluated under the identical methodology.

I report what the measurements support and no more: a score with its 95% interval for every model, which differences from the leading model pass a significance test, what each model cost per question as billed, and how fast it streamed. I do not combine these into a single index.

Each model answered the same 80 ten-option multiple-choice questions (50 from SuperGPQA, hard subset; 30 from MMLU-Pro) through OpenRouter at temperature 0, reasoning effort "low" where supported and at most 4000 output tokens. Answers were graded by a deterministic extraction rule against the datasets' answer keys. Models were tested between October 25, 2026 and October 27, 2026. Chance accuracy is 10%.

meta-llama/llama-4-maverick (2 host failures) had requests that failed after retries; these are excluded from the denominator. qwen/qwen3.7-plus (1), inclusionai/ling-3.1-flash (1), meta-llama/llama-4-scout (1), google/gemma-4-31b-it (1), google/gemma-4-26b-a4b-it (2) and rekaai/reka-flash-3 (1) had answers excluded because the run manifest did not confirm their proof hash (methodology, Sections 7 and 9). Scores below are therefore out of the number of graded answers shown.

Scores range from 1.3% to 75.0%, with a median of 48.8%. The highest score is google/gemini-3.5-flash-lite with 60 of 80 (75.0%; 95% CI 64.5% to 83.2%). For comparison, the median of the frontier group is 75.0%.

Table 2 tests the leader, google/gemini-3.5-flash-lite, against every other model in the group with a two-proportion z-test on the two scores (pooled standard error). A gap counts only when |z| is at least 1.96, the usual threshold for 95% confidence on a two-sided test. Of the 40 comparisons, 28 gaps are significant and 12 are too close to call. The models too close to call against the leader are deepseek/deepseek-v4.1-flash-1017, openai/gpt-6-luna, deepseek/deepseek-v4-flash-vision-exp, anthropic/claude-haiku-5.5, xiaomi/mimo-v2.6-pro, thinkingmachines/inkling-small, z-ai/glm-5.3, z-ai/glm-5.3-flash, minimax/minimax-m3.1-flash, qwen/qwen3.8-27b, qwen/qwen3-coder-next and xiaomi/mimo-v2.6-flash. On this suite their order relative to the leader is unresolved, not a ranking.

As a paired check, an exact McNemar test with Holm-Bonferroni correction (last two columns) agrees with the z-test verdict in 34 of 40 comparisons. It does not confirm the z-test's verdict for qwen/qwen3.7-plus, meta-llama/llama-4-maverick, openai/gpt-oss-120b, tencent/hy3, inception/mercury-2.5 and minimax/minimax-m3, so those gaps should be read with extra caution. The terms are explained below and in the methodology paper.

Two-proportion z-test (opens Wikipedia in a new tab)

Tests whether two success rates differ by more than chance would explain. The gap between the two scores is divided by its standard error (pooled from both models); |z| >= 1.96 corresponds to 95% confidence, two-sided.

McNemar's test (opens Wikipedia in a new tab)

A test for paired yes/no outcomes. Because both models answered the same questions, it looks only at the questions where exactly one of them was right and asks whether that split could be a coin flip. It ignores questions both got right or both got wrong.

Holm-Bonferroni correction (opens Wikipedia in a new tab)

When one model is compared with many others, some comparisons look significant by luck alone. Holm's method raises the bar step by step so that the chance of any false "significant" result across the whole set stays at 5%.

Categories hold between 6 and 30 questions each, so per-category results are shown as counts and are noisy; they indicate where models differ, not reliable sub-rankings.

Billed cost per question ranges from $0.00005 (ibm-granite/granite-4.0-h-micro) to $0.0043, with a median of $0.00062. Spearman's rank correlation between score and cost per question is 0.18: within this group, paying more per question is only weakly related to a higher score.

8 models are on the cost-accuracy Pareto set, from cheapest to most accurate: ibm-granite/granite-4.0-h-micro (25.0%, $0.00005), openai/gpt-oss-20b (51.2%, $0.00007), inception/mercury-2.5 (57.5%, $0.00011), openai/gpt-oss-120b (58.8%, $0.00013), z-ai/glm-5.3-flash (67.5%, $0.00015), openai/gpt-6-luna (72.5%, $0.00025), deepseek/deepseek-v4.1-flash-1017 (73.8%, $0.00091) and google/gemini-3.5-flash-lite (75.0%, $0.0034). Every other model in the group is matched or beaten on both score and cost by at least one of these. Because scores carry the uncertainty shown in Table 1, models just off the set may be equivalent in practice.

Under the 10% rule, no other model comes within 10% of ibm-granite/granite-4.0-h-micro's cost per question.

Spearman's rank correlation (opens Wikipedia in a new tab)

Measures how consistently one quantity rises with another using only their rank order, from -1 to +1. Here it asks whether more expensive models tend to score higher.

Pareto set (cost-accuracy frontier) (opens Wikipedia in a new tab)

The models for which no other model is both at least as accurate and at least as cheap. Any model off the set is beaten on both counts by one on it.

Median output speed across the group is 103 tokens per second and median time to first token is 0.65 s. The highest median output speed is nvidia/nemotron-3.5-super-vl at 715 tokens per second; the lowest median time to first token is mistralai/ministral-3b-2512 at 0.30 s. Under the 10% rule, no other model comes within 10% of nvidia/nemotron-3.5-super-vl's output speed, and mistralai/ministral-8b-2512 is similar to mistralai/ministral-3b-2512 in time to first token. Speed was measured from one client machine; absolute latencies depend on network distance to the host.

20 models returned no extractable answer on at least 8 of the graded questions: google/gemma-4-26b-a4b-it (66), rekaai/reka-flash-3 (66), google/gemma-4-31b-it (50), arcee-ai/trinity-large-thinking (50), mistralai/mistral-small-2603 (44), nvidia/nemotron-3.5-super-vl (43), inclusionai/ling-3.1-flash (40), upstage/solar-mini4 (40), qwen/qwen3.6-35b-a3b (40), upstage/solar-pro4 (39), meituan/longcat-2.0 (34), nvidia/nemotron-3.5-lightning (29), qwen/qwen3.7-plus (23), minimax/minimax-m3 (22), tencent/hy3 (21), xiaomi/mimo-v2.6-flash (18), xiaomi/mimo-v2.6-pro (14), qwen/qwen3.8-27b (14), deepseek/deepseek-v4.1-flash-1017 (12) and minimax/minimax-m3.1-flash (9). Across all models in both groups, 927 of the 939 missing answers ended with finish reason "length" and 932 had an empty answer text, which points to the 4000-token limit being spent on reasoning rather than to wrong answers. These models' scores measure performance under this budget; with a larger budget they may score higher. I do not adjust for this.

The main result is a limit on what can be claimed. With 80 questions, 12 models in this group are too close to call against the leader, so a single ordered leaderboard would overstate what was measured. The data support coarser statements: which models sit clearly below the leader (Table 2), which models give the best accuracy for their billed cost (the Pareto set), and where low scores come from missing answers under a fixed token budget rather than from wrong answers.

Readers choosing a model for production work should weigh these results against task fit: the suite measures expert-level multiple-choice knowledge and reasoning, not coding, tool use or long-context work.

• The group boundary is a list-price rule; it does not imply that every model above it is more capable than every model below it.
• 80 questions in one task format; intervals are roughly +/-10 points.
• One reasoning setting and a 4000-token limit for all models.
• OpenRouter routing may serve a model from different hosts; prices, hosts and model versions change after the test dates.
• Speed and latency come from a single client machine.

Per-model results (llm-evaluation-models.csv) and per-question evidence with proof hashes and generation IDs (llm-evaluation-evidence.csv, llm-evaluation-evidence.json) are published at suhasbhairav.com/research under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Every table in this paper can be recomputed from those files.