ResearchLLM Verifications Leaderboard

Data as of · 78 models, 80 questions · reviewed by Suhas Bhairav · methodology

About

About LLM Verifications Leaderboard

The LLM Verifications Leaderboard, run by Suhas Bhairav: accuracy, output speed (tokens/s), latency and billed API cost for GPT, Claude, Gemini and more, from requests recorded and checked one by one.

Written and maintained by Suhas Bhairav · Data as of

What the leaderboard is

LLM Verifications Leaderboard is an independent research project by Suhas Bhairavthat benchmarks large language models. He sends the same questions and the same speed test to every model through a public API, records every request and response, and publishes what he measured: how often each model is right, how fast it writes, how long it takes to start, and what it actually costs. Every ranking, chart and comparison in the leaderboard is computed from those recordings. Nothing is copied from a vendor's announcement or model card.

The project exists because the numbers that usually decide which model a developer picks are hard to compare. Labs report results on different benchmarks, with different prompts and settings, often picked to show their own model at its best. List prices say little about what a question really costs once a model spends hundreds of tokens thinking. Speed claims depend on who is serving the model and where. He wanted one place where every model is measured the same way, with the uncertainty stated, and the measurements kept as evidence.

What it measures

The latest data covers 78 AI language models from 28 labs, served by 70 API hosts, with the most recent run on 27 Oct 2026. For each model the leaderboard records:

  • Verification score: the share of 80 ten-option multiple-choice questions it answers correctly, drawn from SuperGPQA and MMLU-Pro. These are expert-level questions in subjects such as science, engineering, medicine, economics.
  • Output speed: tokens generated per second once the answer starts streaming, the median of repeated runs of the same prompt.
  • Latency: time to the first token, including any time the model spends thinking first.
  • Cost: what the API actually billed per question, reasoning tokens included, shown next to the list price.
  • Specifications: context window, release date, open or closed weights, and the hosts that serve the model, from OpenRouter's catalogue.

Who runs it

The leaderboard is built and run by Suhas Bhairav, as part of his applied AI research. He is credited as the author and reviewer of every page, and is responsible for everything the leaderboard publishes. In practice, he:

  • Runs the benchmark harness. He chooses which models to test, starts each run, and keeps the proof files, run manifests and harness versions that back every number.
  • Reviews every run before it is published. Each run is checked for host failures, incomplete suites, unusual speed results such as burst delivery, and billed costs that don't match list prices. A run with problems is fixed and re-run rather than edited by hand.
  • Maintains the question suite. He selects the open datasets, samples the questions with fixed seeds, and records the suite's hash so a change of questions is always visible.
  • Maintains the methodology. When the way something is measured changes, the methodology page and the changelog are updated with it.
  • Handles corrections. Reports of errors go to him; he checks the underlying records and publishes a correction where one is needed.

The rules he works by, on independence, sourcing, review and corrections, are set out in the editorial standards.

How it tests

Every model is called through OpenRouter's public API with the same questions and settings: temperature 0, reasoning effort "low" for models that support it. Answers are graded automatically against each dataset's answer key, with no human or AI judge, so a verdict can be re-checked from the response text. Speed and latency come from 3 streamed runs of a fixed prompt. Cost comes from the amount OpenRouter billed for each request.

With 80 questions, scores carry real statistical uncertainty, so comparison pages only call a winner when the gap is statistically significant. Requests that fail on the host's side are retried and, if they still fail, left out of the score instead of being counted as wrong. Every request and response is written to its own file and hashed. The full procedure, with a sample question and the live hashes, is on the methodology page; the source datasets are described on the benchmarks page.

Independence and funding

Rankings come only from the leaderboard's own measurements. The leaderboard does not accept payment from model makers or API hosts for placement, ranking, inclusion or coverage, and there are no sponsored results. The leaderboard doesn't accept model submissions: Suhas decides which models to test, based on what is available on OpenRouter and what readers are likely to compare. The API calls are paid for at normal public prices, the same as any other customer. No lab or host has early access to results or a chance to review them before they are published.

What the numbers are good for, and what they aren't

Use them to shortlist. The verification score is a fair, like-for-like measure of how well models handle hard expert-level questions, and the billed cost and measured speed show the practical trade-offs between models in a way list prices and vendor claims can't. The score-versus-cost chart is a quick way to find models that give the most accuracy for the money.

Be careful with them in a few cases:

  • Small gaps. Two models a few points apart are statistically tied on this many questions. Treat them as equals and decide on price, speed or your own tests.
  • Your own task. Multiple-choice knowledge questions don't measure coding agents, long documents, writing quality, tool use or safety behaviour. Always test the shortlisted models on your own prompts.
  • Your own setup. Speed and latency depend on the host and on your distance to it, and prices change. The figures are a snapshot from a single location on the test date shown on each page.

Where to start

  • The leaderboard lists every model with every measure in one sortable table. Start here if you want the full picture, or sort by cost or speed to find the cheapest or fastest model that clears the accuracy you need.
  • Comparisons put two models side by side, with a written verdict that only names a winner when the difference is larger than chance, and a breakdown by subject.
  • Model pages show one model in depth: its score and rank, speed runs, billed cost against list price, the hosts that serve it, and the date it was last tested.
  • Provider pages group the models by the lab that makes them.
  • Subject pages break the score down by field, such as science, engineering, medicine or law, with the caveat that each subject has only a few questions.
  • The price calculator turns per-token prices into a monthly bill for your own traffic, and the FAQ answers the most common questions in a sentence or two.

Every data page shows the date its numbers were measured. Results change as models are updated and re-tested, so when you make a decision, check the date and, for close calls, re-check the page before you commit.

How to cite the leaderboard

You are welcome to quote and cite the leaderboard's results. Please link to the page you took the number from and give the test date shown there. A suggested citation:

LLM Verifications Leaderboard (2026). AI model benchmark results. https://suhasbhairav.com/research/llm-leaderboard/leaderboard (data as of 27 Oct 2026).

Journalists and researchers will find facts and a short description of the leaderboard on the media page. You may reuse the data with a link back, as set out on the downloads page; the source questions remain under their own licences.

Corrections and contact

If you think a number is wrong, let Suhas know which page and which model, and what you expected. Suhas checks the recorded requests and responses behind it. If he got something wrong, he fixes the cause, re-runs the affected tests and notes the change in the changelog.

Email suhasbhairav@gmail.com, or book a call.