ResearchLLM Verifications Leaderboard

Data as of · 78 models, 80 questions · reviewed by Suhas Bhairav · methodology

Head to head · tested 27 Oct 2026

Nemotron 3.5 Lightning vs Phi 4: accuracy, speed and price

Similar accuracy; Phi 4 costs less per question.

NVIDIANemotron 3.5 Lightning43.8% score · #51$0.00049 per question116 tok/sMicrosoftPhi 441.2% score · #54$0.00012 per question103 tok/s

Verdict

Built from the measurements below

  • Accuracy is too close to call: Nemotron 3.5 Lightning scored 43.8% and Phi 4 41.2% on the same 80 questions, a gap within what chance alone could produce.
  • Phi 4 is 4.1× cheaper per question as billed ($0.00012 vs $0.00049), reasoning tokens included.
  • Nemotron 3.5 Lightning generates text 1.1× faster: 116 tokens per second against 103.
  • Nemotron 3.5 Lightning accepts more context: 262k tokens against 16k.

Which should you choose?

Choose Nemotron 3.5 Lightning for faster streaming, a 262k-token context window and more hosts to choose from on OpenRouter (4 against 1). Choose Phi 4 for the lower cost per correct answer ($0.00029 against $0.0011) and shorter outputs (about 699 tokens per question against 2,846). The widest subject gap is in humanities & social science: Nemotron 3.5 Lightning answered 6 of 9 correctly, Phi 4 4 of 9. With so few questions per subject, treat this as a lead to test on your own prompts, not a result.

Accuracy gap +2.5 points (z = 0.32). A winner is called only when z is at least 1.96, the usual 95% threshold. Speed, latency and price need a 10% margin.

Head to head

Same questions, same prompt, same API

MeasureNemotron 3.5 LightningPhi 4Edge
Verification scoreHigher is better43.8%41.2%Too close to call
Cost per questionLower is better$0.00049$0.00012Phi 4
Output speedHigher is better116103Nemotron 3.5 Lightning
Time to first tokenLower is better0.5 s0.5 sSimilar
End-to-end response timeLower is better6.5 s4.9 sPhi 4
Tokens per questionLower is better2,846699Phi 4
Context windowHigher is better262k16kNemotron 3.5 Lightning

Side by side

Bars are scaled to the larger of the two values. The winner on each measure is marked in bold.

Every measure

Nemotron 3.5 Lightning first, Phi 4 second

Verification scorehigher is better
Nemotron 3.5 Lightning43.8%
Phi 441.2%
Cost per questionlower is better
Nemotron 3.5 Lightning$0.00049
Phi 4$0.00012
Output speedhigher is better
Nemotron 3.5 Lightning116
Phi 4103
Time to first tokenlower is better
Nemotron 3.5 Lightning0.5 s
Phi 40.5 s
End-to-end response timelower is better
Nemotron 3.5 Lightning6.5 s
Phi 44.9 s
Tokens per questionlower is better
Nemotron 3.5 Lightning2,846
Phi 4699
Context windowhigher is better
Nemotron 3.5 Lightning262k
Phi 416k

Score vs. cost per question

Both models in color, every other model in grey

Most attractive quadrant0255075100$0.00001$0.0001$0.001$0.01$0.10Cost per question (USD, log) →Verification score (%) →Qwen3.8 2.4T A95BCommand AReka Flash 3Gemma 4 26B A4BGemma 4 31BQwen3.5 397B A17BHermes 4 405BMuse WonderGrok 4.7o3gpt-oss-20bMinistral 3 3B 2512Nova Micro 1.0Mercury 2.5Phi 4Nemotron 3.5 LightningGranite 4.2 8BHy3GLM 5.3 FlashTrinity Large ThinkingMiniMax M3Mistral Large 4Claude Opus 5.5GPT-6.1 SolGemini 4 ArgonGPT-5.4GPT-6 Lunagpt-oss-120b

Specifications and pricing

From OpenRouter's catalogue, plus the leaderboard's test dates

SpecificationNemotron 3.5 LightningPhi 4
CreatorNVIDIAMicrosoft
Released11 Aug 202610 Jan 2025
WeightsOpen weightsOpen weights
ReasoningYesNo
Inputtexttext
Outputtexttext
Context window262k tokens16k tokens
Knowledge cutoff—30 Jun 2024
List price in / out per 1M$0.07 / $0.2$0.07 / $0.14
Blended price per 1M (3:1)$0.10$0.09
Cheapest hostDarkbloom · $0.025 / $0.12DeepInfra · $0.056 / $0.112
Hosts on OpenRouter41
Last tested26 Oct 202626 Oct 2026

Compare any two models →

Nemotron 3.5 Lightning vs Phi 4: frequently asked questions

Is Nemotron 3.5 Lightning better than Phi 4?
On accuracy they are statistically tied: Nemotron 3.5 Lightning scored 43.8% and Phi 4 41.2% on the same 80 questions from open datasets. Similar accuracy; Phi 4 costs less per question.
Which is cheaper, Nemotron 3.5 Lightning or Phi 4?
Phi 4 is cheaper: $0.00012 per question as billed, against $0.00049. List prices: Nemotron 3.5 Lightning $0.07 in / $0.2 out, Phi 4 $0.07 in / $0.14 out per million tokens.
Which is faster, Nemotron 3.5 Lightning or Phi 4?
Nemotron 3.5 Lightning is faster at 116 output tokens per second, against 103. Time to first token: 0.5 s for Nemotron 3.5 Lightning, 0.5 s for Phi 4.
Which has the larger context window, Nemotron 3.5 Lightning or Phi 4?
Nemotron 3.5 Lightning, with 262k tokens against 16k.