ResearchLLM Verifications Leaderboard

Data as of · 78 models, 80 questions · reviewed by Suhas Bhairav · methodology

Head to head · tested 27 Oct 2026

gpt-oss-20b vs Mercury 2.5: accuracy, speed and price

Similar accuracy; gpt-oss-20b costs less per question.

OpenAIgpt-oss-20b51.2% score · #46$0.000066 per question103 tok/sInceptionMercury 2.557.5% score · #43$0.00011 per question157 tok/s

Verdict

Built from the measurements below

  • Accuracy is too close to call: gpt-oss-20b scored 51.2% and Mercury 2.5 57.5% on the same 80 questions, a gap within what chance alone could produce.
  • gpt-oss-20b is 1.7× cheaper per question as billed ($0.000066 vs $0.00011), reasoning tokens included.
  • Mercury 2.5 generates text 1.5× faster: 157 tokens per second against 103.
  • gpt-oss-20b starts answering sooner: 0.7 s to the first token, against 2.3 s for Mercury 2.5.
  • Mercury 2.5 accepts more context: 260k tokens against 131k.

Which should you choose?

Choose gpt-oss-20b for the lower cost per correct answer ($0.00013 against $0.00019), a quicker first token, open weights you can run yourself and more hosts to choose from on OpenRouter (10 against 1). Choose Mercury 2.5 for faster streaming and a 260k-token context window. The widest subject gap is in economics: Mercury 2.5 answered 8 of 9 correctly, gpt-oss-20b 6 of 9. With so few questions per subject, treat this as a lead to test on your own prompts, not a result.

Accuracy gap -6.3 points (z = -0.79). A winner is called only when z is at least 1.96, the usual 95% threshold. Speed, latency and price need a 10% margin.

Head to head

Same questions, same prompt, same API

Measuregpt-oss-20bMercury 2.5Edge
Verification scoreHigher is better51.2%57.5%Too close to call
Cost per questionLower is better$0.000066$0.00011gpt-oss-20b
Output speedHigher is better103157Mercury 2.5
Time to first tokenLower is better0.7 s2.3 sgpt-oss-20b
End-to-end response timeLower is better4.0 s6.5 sgpt-oss-20b
Tokens per questionLower is better472673gpt-oss-20b
Context windowHigher is better131k260kMercury 2.5

Side by side

Bars are scaled to the larger of the two values. The winner on each measure is marked in bold.

Every measure

gpt-oss-20b first, Mercury 2.5 second

Verification scorehigher is better
gpt-oss-20b51.2%
Mercury 2.557.5%
Cost per questionlower is better
gpt-oss-20b$0.000066
Mercury 2.5$0.00011
Output speedhigher is better
gpt-oss-20b103
Mercury 2.5157
Time to first tokenlower is better
gpt-oss-20b0.7 s
Mercury 2.52.3 s
End-to-end response timelower is better
gpt-oss-20b4.0 s
Mercury 2.56.5 s
Tokens per questionlower is better
gpt-oss-20b472
Mercury 2.5673
Context windowhigher is better
gpt-oss-20b131k
Mercury 2.5260k

Score vs. cost per question

Both models in color, every other model in grey

Most attractive quadrant0255075100$0.00001$0.0001$0.001$0.01$0.10Cost per question (USD, log) →Verification score (%) →Qwen3.8 2.4T A95BCommand AReka Flash 3Gemma 4 26B A4BGemma 4 31BQwen3.5 397B A17BHermes 4 405BMuse WonderGrok 4.7o3gpt-oss-20bMinistral 3 3B 2512Nova Micro 1.0Mercury 2.5Phi 4Granite 4.2 8BHy3GLM 5.3 FlashSolar Pro 4Trinity Large ThinkingMiniMax M3Mistral Large 4Claude Opus 5.5GPT-6.1 SolGemini 4 ArgonGPT-5.4GPT-6 Lunagpt-oss-120b

Specifications and pricing

From OpenRouter's catalogue, plus the leaderboard's test dates

Specificationgpt-oss-20bMercury 2.5
CreatorOpenAIInception
Released5 Aug 20258 Sept 2026
WeightsOpen weights—
ReasoningYesYes
Inputtexttext
Outputtexttext
Context window131k tokens260k tokens
Knowledge cutoff30 Jun 2024—
List price in / out per 1M$0.018 / $0.09$0.04 / $0.15
Blended price per 1M (3:1)$0.04$0.07
Cheapest hostDarkbloom · $0.018 / $0.09Inception · $0.04 / $0.15
Hosts on OpenRouter101
Last tested26 Oct 202626 Oct 2026

Compare any two models →

gpt-oss-20b vs Mercury 2.5: frequently asked questions

Is gpt-oss-20b better than Mercury 2.5?
On accuracy they are statistically tied: gpt-oss-20b scored 51.2% and Mercury 2.5 57.5% on the same 80 questions from open datasets. Similar accuracy; gpt-oss-20b costs less per question.
Which is cheaper, gpt-oss-20b or Mercury 2.5?
gpt-oss-20b is cheaper: $0.000066 per question as billed, against $0.00011. List prices: gpt-oss-20b $0.018 in / $0.09 out, Mercury 2.5 $0.04 in / $0.15 out per million tokens.
Which is faster, gpt-oss-20b or Mercury 2.5?
Mercury 2.5 is faster at 157 output tokens per second, against 103. Time to first token: 0.7 s for gpt-oss-20b, 2.3 s for Mercury 2.5.
Which has the larger context window, gpt-oss-20b or Mercury 2.5?
Mercury 2.5, with 260k tokens against 131k.