ResearchLLM Verifications Leaderboard

Data as of · 78 models, 80 questions · reviewed by Suhas Bhairav · methodology

Head to head · tested 27 Oct 2026

Mercury 2.5 vs Nemotron 3.5 Lightning: accuracy, speed and price

Similar accuracy; Mercury 2.5 costs less per question.

InceptionMercury 2.557.5% score · #43$0.00011 per question157 tok/sNVIDIANemotron 3.5 Lightning43.8% score · #51$0.00049 per question116 tok/s

Verdict

Built from the measurements below

  • Accuracy is too close to call: Mercury 2.5 scored 57.5% and Nemotron 3.5 Lightning 43.8% on the same 80 questions, a gap within what chance alone could produce.
  • Mercury 2.5 is 4.4× cheaper per question as billed ($0.00011 vs $0.00049), reasoning tokens included.
  • Mercury 2.5 generates text 1.4× faster: 157 tokens per second against 116.
  • Nemotron 3.5 Lightning starts answering sooner: 0.5 s to the first token, against 2.3 s for Mercury 2.5.

Which should you choose?

Choose Mercury 2.5 for the lower cost per correct answer ($0.00019 against $0.0011), faster streaming and shorter outputs (about 673 tokens per question against 2,846). Choose Nemotron 3.5 Lightning for a quicker first token, open weights you can run yourself and more hosts to choose from on OpenRouter (4 against 1). The widest subject gap is in law: Mercury 2.5 answered 3 of 6 correctly, Nemotron 3.5 Lightning 1 of 6. With so few questions per subject, treat this as a lead to test on your own prompts, not a result.

Accuracy gap +13.7 points (z = 1.74). A winner is called only when z is at least 1.96, the usual 95% threshold. Speed, latency and price need a 10% margin.

Head to head

Same questions, same prompt, same API

MeasureMercury 2.5Nemotron 3.5 LightningEdge
Verification scoreHigher is better57.5%43.8%Too close to call
Cost per questionLower is better$0.00011$0.00049Mercury 2.5
Output speedHigher is better157116Mercury 2.5
Time to first tokenLower is better2.3 s0.5 sNemotron 3.5 Lightning
End-to-end response timeLower is better6.5 s6.5 sSimilar
Tokens per questionLower is better6732,846Mercury 2.5
Context windowHigher is better260k262kSimilar

Side by side

Bars are scaled to the larger of the two values. The winner on each measure is marked in bold.

Every measure

Mercury 2.5 first, Nemotron 3.5 Lightning second

Verification scorehigher is better
Mercury 2.557.5%
Nemotron 3.5 Lightning43.8%
Cost per questionlower is better
Mercury 2.5$0.00011
Nemotron 3.5 Lightning$0.00049
Output speedhigher is better
Mercury 2.5157
Nemotron 3.5 Lightning116
Time to first tokenlower is better
Mercury 2.52.3 s
Nemotron 3.5 Lightning0.5 s
End-to-end response timelower is better
Mercury 2.56.5 s
Nemotron 3.5 Lightning6.5 s
Tokens per questionlower is better
Mercury 2.5673
Nemotron 3.5 Lightning2,846
Context windowhigher is better
Mercury 2.5260k
Nemotron 3.5 Lightning262k

Score vs. cost per question

Both models in color, every other model in grey

Most attractive quadrant0255075100$0.00001$0.0001$0.001$0.01$0.10Cost per question (USD, log) →Verification score (%) →Qwen3.8 2.4T A95BCommand AReka Flash 3Gemma 4 26B A4BGemma 4 31BQwen3.5 397B A17BHermes 4 405BMuse WonderGrok 4.7o3gpt-oss-20bMinistral 3 3B 2512Nova Micro 1.0Mercury 2.5Phi 4Nemotron 3.5 LightningGranite 4.2 8BHy3GLM 5.3 FlashTrinity Large ThinkingMiniMax M3Mistral Large 4Claude Opus 5.5GPT-6.1 SolGemini 4 ArgonGPT-5.4GPT-6 Lunagpt-oss-120b

Specifications and pricing

From OpenRouter's catalogue, plus the leaderboard's test dates

SpecificationMercury 2.5Nemotron 3.5 Lightning
CreatorInceptionNVIDIA
Released8 Sept 202611 Aug 2026
Weights—Open weights
ReasoningYesYes
Inputtexttext
Outputtexttext
Context window260k tokens262k tokens
Knowledge cutoff——
List price in / out per 1M$0.04 / $0.15$0.07 / $0.2
Blended price per 1M (3:1)$0.07$0.10
Cheapest hostInception · $0.04 / $0.15Darkbloom · $0.025 / $0.12
Hosts on OpenRouter14
Last tested26 Oct 202626 Oct 2026

Compare any two models →

Mercury 2.5 vs Nemotron 3.5 Lightning: frequently asked questions

Is Mercury 2.5 better than Nemotron 3.5 Lightning?
On accuracy they are statistically tied: Mercury 2.5 scored 57.5% and Nemotron 3.5 Lightning 43.8% on the same 80 questions from open datasets. Similar accuracy; Mercury 2.5 costs less per question.
Which is cheaper, Mercury 2.5 or Nemotron 3.5 Lightning?
Mercury 2.5 is cheaper: $0.00011 per question as billed, against $0.00049. List prices: Mercury 2.5 $0.04 in / $0.15 out, Nemotron 3.5 Lightning $0.07 in / $0.2 out per million tokens.
Which is faster, Mercury 2.5 or Nemotron 3.5 Lightning?
Mercury 2.5 is faster at 157 output tokens per second, against 116. Time to first token: 2.3 s for Mercury 2.5, 0.5 s for Nemotron 3.5 Lightning.
Which has the larger context window, Mercury 2.5 or Nemotron 3.5 Lightning?
Both accept about the same amount of context: 260k and 262k tokens.