ResearchLLM Verifications Leaderboard

Data as of · 78 models, 80 questions · reviewed by Suhas Bhairav · methodology

Head to head · tested 27 Oct 2026

Gemma 4 31B vs Mistral Small 4: accuracy, speed and price

Similar accuracy; Gemma 4 31B costs less per question.

GoogleGemma 4 31B30.0% score · #64$0.0014 per question58 tok/sMistralMistral Small 432.5% score · #62$0.0019 per question170 tok/s

Verdict

Built from the measurements below

  • Accuracy is too close to call: Gemma 4 31B scored 30.0% and Mistral Small 4 32.5% on the same 80 questions, a gap within what chance alone could produce.
  • Gemma 4 31B is 1.3× cheaper per question as billed ($0.0014 vs $0.0019), reasoning tokens included.
  • Mistral Small 4 generates text 2.9× faster: 170 tokens per second against 58.
  • Mistral Small 4 starts answering sooner: 0.5 s to the first token, against 0.6 s for Gemma 4 31B.

Which should you choose?

Choose Gemma 4 31B for the lower cost per correct answer ($0.0046 against $0.0057) and more hosts to choose from on OpenRouter (13 against 1). Choose Mistral Small 4 for faster streaming and a quicker first token. The widest subject gap is in medicine: Mistral Small 4 answered 5 of 11 correctly, Gemma 4 31B 2 of 11. With so few questions per subject, treat this as a lead to test on your own prompts, not a result.

Accuracy gap -2.5 points (z = -0.34). A winner is called only when z is at least 1.96, the usual 95% threshold. Speed, latency and price need a 10% margin.

Head to head

Same questions, same prompt, same API

MeasureGemma 4 31BMistral Small 4Edge
Verification scoreHigher is better30.0%32.5%Too close to call
Cost per questionLower is better$0.0014$0.0019Gemma 4 31B
Output speedHigher is better58170Mistral Small 4
Time to first tokenLower is better0.6 s0.5 sMistral Small 4
End-to-end response timeLower is better12.6 s4.5 sMistral Small 4
Tokens per questionLower is better3,3073,021Similar
Context windowHigher is better262k262kSimilar

Side by side

Bars are scaled to the larger of the two values. The winner on each measure is marked in bold.

Every measure

Gemma 4 31B first, Mistral Small 4 second

Verification scorehigher is better
Gemma 4 31B30.0%
Mistral Small 432.5%
Cost per questionlower is better
Gemma 4 31B$0.0014
Mistral Small 4$0.0019
Output speedhigher is better
Gemma 4 31B58
Mistral Small 4170
Time to first tokenlower is better
Gemma 4 31B0.6 s
Mistral Small 40.5 s
End-to-end response timelower is better
Gemma 4 31B12.6 s
Mistral Small 44.5 s
Tokens per questionlower is better
Gemma 4 31B3,307
Mistral Small 43,021
Context windowhigher is better
Gemma 4 31B262k
Mistral Small 4262k

Score vs. cost per question

Both models in color, every other model in grey

Most attractive quadrant0255075100$0.00001$0.0001$0.001$0.01$0.10Cost per question (USD, log) →Verification score (%) →Qwen3.8 2.4T A95BCommand AReka Flash 3Gemma 4 26B A4BGemma 4 31BQwen3.5 397B A17BHermes 4 405BMuse WonderGrok 4.7o3gpt-oss-20bMinistral 3 3B 2512Nova Micro 1.0Phi 4Granite 4.2 8BHy3GLM 5.3 FlashSolar Pro 4Trinity Large ThinkingMiniMax M3Mistral Large 4Claude Opus 5.5GPT-6.1 SolGemini 4 ArgonGPT-5.4GPT-6 Lunagpt-oss-120bLlama 4 Maverick

Specifications and pricing

From OpenRouter's catalogue, plus the leaderboard's test dates

SpecificationGemma 4 31BMistral Small 4
CreatorGoogleMistral
Released2 Apr 202616 Mar 2026
WeightsOpen weightsOpen weights
ReasoningYesYes
Inputimage, text, videotext, image
Outputtexttext
Context window262k tokens262k tokens
Knowledge cutoff——
List price in / out per 1M$0.2 / $0.4$0.15 / $0.6
Blended price per 1M (3:1)$0.25$0.26
Cheapest hostDeepInfra · $0.16 / $0.32Mistral · $0.15 / $0.6
Hosts on OpenRouter131
Last tested27 Oct 202626 Oct 2026

Compare any two models →

Gemma 4 31B vs Mistral Small 4: frequently asked questions

Is Gemma 4 31B better than Mistral Small 4?
On accuracy they are statistically tied: Gemma 4 31B scored 30.0% and Mistral Small 4 32.5% on the same 80 questions from open datasets. Similar accuracy; Gemma 4 31B costs less per question.
Which is cheaper, Gemma 4 31B or Mistral Small 4?
Gemma 4 31B is cheaper: $0.0014 per question as billed, against $0.0019. List prices: Gemma 4 31B $0.2 in / $0.4 out, Mistral Small 4 $0.15 in / $0.6 out per million tokens.
Which is faster, Gemma 4 31B or Mistral Small 4?
Mistral Small 4 is faster at 170 output tokens per second, against 58. Time to first token: 0.6 s for Gemma 4 31B, 0.5 s for Mistral Small 4.
Which has the larger context window, Gemma 4 31B or Mistral Small 4?
Both accept about the same amount of context: 262k and 262k tokens.