ResearchLLM Verifications Leaderboard

Data as of · 78 models, 80 questions · reviewed by Suhas Bhairav · methodology

Head to head · tested 27 Oct 2026

Nemotron 3.5 Lightning vs Nemotron 3 Ultra: accuracy, speed and price

Nemotron 3 Ultra is more accurate; Nemotron 3.5 Lightning is cheaper.

NVIDIANemotron 3.5 Lightning43.8% score · #51$0.00049 per question116 tok/sNVIDIANemotron 3 Ultra60.0% score · #38$0.0057 per question136 tok/s

Verdict

Built from the measurements below

  • Nemotron 3 Ultra is more accurate: 60.0% against 43.8% on the same 80 questions, a 16.2-point gap that is statistically significant.
  • Nemotron 3.5 Lightning is 12× cheaper per question as billed ($0.00049 vs $0.0057), reasoning tokens included.
  • Nemotron 3 Ultra generates text 1.2× faster: 136 tokens per second against 116.
  • Nemotron 3.5 Lightning starts answering sooner: 0.5 s to the first token, against 0.8 s for Nemotron 3 Ultra.

Which should you choose?

Choose Nemotron 3.5 Lightning for the lower cost per correct answer ($0.0011 against $0.0094) and a quicker first token. Choose Nemotron 3 Ultra for the most accurate answers of the two and faster streaming. The widest subject gap is in humanities & social science: Nemotron 3 Ultra answered 9 of 9 correctly, Nemotron 3.5 Lightning 6 of 9. With so few questions per subject, treat this as a lead to test on your own prompts, not a result.

Accuracy gap -16.2 points (z = -2.06). A winner is called only when z is at least 1.96, the usual 95% threshold. Speed, latency and price need a 10% margin.

Head to head

Same questions, same prompt, same API

MeasureNemotron 3.5 LightningNemotron 3 UltraEdge
Verification scoreHigher is better43.8%60.0%Nemotron 3 Ultra
Cost per questionLower is better$0.00049$0.0057Nemotron 3.5 Lightning
Output speedHigher is better116136Nemotron 3 Ultra
Time to first tokenLower is better0.5 s0.8 sNemotron 3.5 Lightning
End-to-end response timeLower is better6.5 s3.6 sNemotron 3 Ultra
Tokens per questionLower is better2,8462,177Nemotron 3 Ultra
Context windowHigher is better262k262kSimilar

Side by side

Bars are scaled to the larger of the two values. The winner on each measure is marked in bold.

Every measure

Nemotron 3.5 Lightning first, Nemotron 3 Ultra second

Verification scorehigher is better
Nemotron 3.5 Lightning43.8%
Nemotron 3 Ultra60.0%
Cost per questionlower is better
Nemotron 3.5 Lightning$0.00049
Nemotron 3 Ultra$0.0057
Output speedhigher is better
Nemotron 3.5 Lightning116
Nemotron 3 Ultra136
Time to first tokenlower is better
Nemotron 3.5 Lightning0.5 s
Nemotron 3 Ultra0.8 s
End-to-end response timelower is better
Nemotron 3.5 Lightning6.5 s
Nemotron 3 Ultra3.6 s
Tokens per questionlower is better
Nemotron 3.5 Lightning2,846
Nemotron 3 Ultra2,177
Context windowhigher is better
Nemotron 3.5 Lightning262k
Nemotron 3 Ultra262k

Score vs. cost per question

Both models in color, every other model in grey

Most attractive quadrant0255075100$0.00001$0.0001$0.001$0.01$0.10Cost per question (USD, log) →Verification score (%) →Qwen3.8 2.4T A95BCommand AReka Flash 3Gemma 4 26B A4BGemma 4 31BQwen3.5 397B A17BHermes 4 405BMuse WonderGrok 4.7o3gpt-oss-20bMinistral 3 3B 2512Nova Micro 1.0Mercury 2.5Phi 4Nemotron 3.5 LightningGranite 4.2 8BHy3GLM 5.3 FlashTrinity Large ThinkingMiniMax M3Mistral Large 4Claude Opus 5.5GPT-6.1 SolGemini 4 ArgonGPT-5.4GPT-6 Lunagpt-oss-120b

Specifications and pricing

From OpenRouter's catalogue, plus the leaderboard's test dates

SpecificationNemotron 3.5 LightningNemotron 3 Ultra
CreatorNVIDIANVIDIA
Released11 Aug 20264 Jun 2026
WeightsOpen weightsOpen weights
ReasoningYesYes
Inputtexttext
Outputtexttext
Context window262k tokens262k tokens
Knowledge cutoff——
List price in / out per 1M$0.07 / $0.2$0.6 / $2.4
Blended price per 1M (3:1)$0.10$1.05
Cheapest hostDarkbloom · $0.025 / $0.12DeepInfra · $0.4 / $1.76
Hosts on OpenRouter43
Last tested26 Oct 202626 Oct 2026

Compare any two models →

Nemotron 3.5 Lightning vs Nemotron 3 Ultra: frequently asked questions

Is Nemotron 3.5 Lightning better than Nemotron 3 Ultra?
Nemotron 3 Ultra is more accurate: it answered 48 of 80 questions correctly (60.0%) against 35 of 80 (43.8%) for Nemotron 3.5 Lightning. Nemotron 3 Ultra is more accurate; Nemotron 3.5 Lightning is cheaper.
Which is cheaper, Nemotron 3.5 Lightning or Nemotron 3 Ultra?
Nemotron 3.5 Lightning is cheaper: $0.00049 per question as billed, against $0.0057. List prices: Nemotron 3.5 Lightning $0.07 in / $0.2 out, Nemotron 3 Ultra $0.6 in / $2.4 out per million tokens.
Which is faster, Nemotron 3.5 Lightning or Nemotron 3 Ultra?
Nemotron 3 Ultra is faster at 136 output tokens per second, against 116. Time to first token: 0.5 s for Nemotron 3.5 Lightning, 0.8 s for Nemotron 3 Ultra.
Which has the larger context window, Nemotron 3.5 Lightning or Nemotron 3 Ultra?
Both accept about the same amount of context: 262k and 262k tokens.