Independent research · Suhas Bhairav
Applied AI and LLM Research
I run my own experiments on large language models and AI models: how accurate they are, what they really cost and how fast they respond. The results are published as papers with PDFs, and every number traces to a recorded API request and response, so you can check the work instead of taking it on trust.
- Models evaluated
- 72
- Graded answers
- 5,746
- Papers
- 3
Research focus
LLM evaluation
Measuring what large language models actually get right on expert-level questions, with uncertainty reported instead of hidden.
AI model comparison
Accuracy, billed cost and speed for frontier and lower-cost models under identical conditions, so model choices rest on evidence.
Reproducible results
Every published number links to a recorded API request and response, so readers can check the work rather than trust it.
Latest research: independent LLM evaluation
Published . Data collected to from 72 models.
Methodology
How Every Number Is Produced: A Verifiable Methodology for Evaluating Large Language Models
How to evaluate LLMs so every number is checkable: 80 open-dataset questions, deterministic grading, billed costs, Wilson intervals and SHA-256 proofs.
- Questions
- 80
- Proof hashes verified
- 5,760/5,760
- Verdicts re-graded
- 5,756/5,756
Frontier models
Frontier Language Models Under a Verifiable Protocol: Accuracy, Cost and Speed on Expert-Level Questions
31 frontier LLMs on 80 expert questions: scores 26.6%-85.0% with 95% intervals, significance tests, billed cost and speed.
- Models
- 31
- Score range
- 26.6%–85.0%
- Median cost / question
- $0.0088
Non-frontier models
Non-Frontier Language Models Under a Verifiable Protocol: Accuracy, Cost and Speed on Expert-Level Questions
41 non-frontier LLMs on 80 expert questions: scores 1.3%-75.0% with 95% intervals, significance tests, billed cost and speed.
- Models
- 41
- Score range
- 1.3%–75.0%
- Median cost / question
- $0.00062
How the evidence was checked
- 5,760 of 5,760 answer proof files match their SHA-256.
- 5,756 of 5,756 answers re-graded independently give the recorded verdict.
- 72 of 72 model scores and 72 of 72 billed costs recompute from the proofs.
- 12 answers not confirmed by their run manifest and 2 host failures are excluded, leaving 5,746 graded answers.
Data
Licence: CC BY 4.0. Reuse is welcome with credit to Suhas Bhairav.
Suite SHA-256: cd444ee142dc6b87e74967c9925ea9b7b171b90fe0e18b5a214c4587d964121a
Questions and answers
- What research does Suhas Bhairav do?
- Suhas Bhairav does independent applied AI research on large language models: how accurate they are, what they cost and how fast they respond in practice. His current series evaluates 72 LLMs (31 frontier and 41 non-frontier) on the same 80 expert-level questions, with every result published alongside its evidence.
- Which LLMs scored highest in Suhas Bhairav's evaluation?
- Among frontier models, anthropic/claude-opus-5.5 scored highest at 85.0%, but 19 other frontier models are too close to call against it. Among non-frontier models, google/gemini-3.5-flash-lite scored highest at 75.0%, with 12 others too close to call.
- Can the results be checked independently?
- Yes. Per-question verdicts, OpenRouter generation IDs and SHA-256 hashes of every proof file are published as CSV and JSON. 5,760 of 5,760 proof hashes and 5,756 of 5,756 verdicts were re-verified before publication.
- How are frontier and non-frontier models separated?
- By list price: a model is frontier when its blended OpenRouter list price, (3 x input + 1 x output) / 4, is at least $1.00 per million tokens.