The question it answers
"Which model should I ship in Hindi, Kannada, Tamil, Telugu, Malayalam or Hinglish, and what will it cost to run?" AI4Bharat's Indic LLM-Arena already ranks which answers people prefer. This leaderboard measures the tasks a product depends on and the running cost of each language instead.
Models and hardware
All models are open weights, downloaded to an external drive and served through Ollama, mostly as Q4_K_M GGUF quantisations (the table shows each model's quantisation). Kalike, a nanochat model that Ollama cannot load, runs in its own PyTorch environment. Runs use one Apple M3 Max, 64 GBwith one model in memory at a time. Models that reason before answering run with reasoning off where they allow it; gpt-oss cannot turn it off and runs at low effort with a larger token budget. API models (Claude, GPT, Gemini, Sarvam's hosted API) are not in this run.
Tasks
- Retrieval Q&A (product): Answer a customer's question from a short policy or product passage, or say NOT_FOUND when the passage has no answer.
- Tool calling (product): Turn a user's request into the right function call with the right arguments, as JSON.
- Support routing (product): Route a support message to one of six categories.
- Support reply (product): Write a short apology and fix for an upset customer, in their language. Judged with a rubric.
- Summarization (product): Summarise a local news item in two sentences, keeping the facts. Judged with a rubric.
- MMLU-Indic (knowledge): Sarvam's Indic translation of MMLU (general knowledge, 4 options). Source: MMLU-Indic (Sarvam) (
sarvamai/mmlu-indic), licence: MIT. - IndicMMLU-Pro (knowledge): IndicMMLU-Pro: harder reasoning questions with up to 10 options. Source: IndicMMLU-Pro (
LinguaLift/IndicMMLU-Pro), licence: see dataset card (no license field). - MILU (knowledge): AI4Bharat MILU: India-specific knowledge across 41 subjects. Source: MILU (AI4Bharat) (
ai4bharat/MILU), licence: AI4Bharat terms, see dataset card.
The product score is the mean of the five product tasks, each weighted equally. The knowledge score is the mean of the public multiple-choice benchmarks available for that language (100 MMLU-Indic and 60IndicMMLU-Pro questions per language, a seeded sample stratified by subject). IndicMMLU-Pro has no Malayalam set. For Hinglish the knowledge score uses Sarvam's romanised Hindi MMLU. IndicMMLU-Pro questions that were only blanks after translation are dropped.
The private test set
The product tasks come from a private set of 17 parallel scenarios per language (version v0-2026-11-22): the same facts and answers in every language, so scores and token counts compare like for like. Companies, prices and people in it are fictional. It is not published, so models cannot be trained on the test; the datasets page describes it and links every public benchmark.
Status: draft, pending native-speaker review. Version 0 was drafted with an AI assistant (Claude) as a starting point and has not yet been checked by native speakers of every language. It is small, so differences of a few points are within noise: each task shows its item count (n) and 95% interval in the downloadable data. The target is 200–500 items per language, written or checked by native speakers.
Prompts
Every prompt is a single user turn. Instructions are in English, as in most Indian products; the content the end user would type or read (the passage, the message, the question) is in the target language, and the model is asked to answer in that language. This is the same for every model, which disadvantages models that only understand one Indic language, such as Kalike.
Scoring
- Multiple choice: exact match on the option letter. The model has 16 tokens to give it; a model that starts explaining instead of answering is marked wrong, the same rule for every model. Machine translation has damaged some IndicMMLU-Pro questions (in Hindi, for example, some options that differ in English read identically), so its scores are noisy and lower than the English original for every model; compare models within a language rather than across.
- Retrieval Q&A: the answer's opening must contain the fact (native digits and number words count) and must not say NOT_FOUND; the unanswerable item passes only with NOT_FOUND.
- Tool calling: valid JSON, the right function, and every argument correct (phone digits, amounts and city spellings normalised).
- Support routing: exactly one category, the right one.
- Support reply and summary: a local LLM judge (gemma4-31b, via Ollama) grades language, faithfulness, task and fluency from 1 to 5 with a fixed rubric; the mean is scaled to 0–100. The judge is also ranked, so its own answers are flagged as self-judged in the data, and 10% of verdicts are exported for native-speaker spot checks.
- Answers in language: share of free-text answers whose letters are at least 60% in the target script (Latin for Hinglish).
Tokens, speed, memory and rupees
Tokens per word and the multiplier against Englishare measured on the private set's parallel passages with each model's own tokenizer. Tok/s is the median output speed on answers of 40 tokens or more; first token is the median time to the first visible token, with the model already in memory. Memory is what Ollama reports for the loaded model (unified memory on a Mac, the equivalent of VRAM).
₹ per 1,000 words is the electricity to write 1,000 words in that language on the test machine: seconds per 1,000 words × 90 W × ₹8/kWh. It is an approximation, and it leaves out the hardware itself; its value is in comparing models and languages. API prices in rupees will be added when API models are.
Conflict of interest
Suhas Bhairav runs this leaderboard alone and built one of the models on it, Kalike. Kalike is scored exactly like the others and labelled wherever it appears. Nothing in the test set was used to train or tune it.
Reproducing a run
The harness is a Python project: python main.py pull, run, judge, build. Each run stores, per request, the model's exact Ollama digest, the prompt and its hash, the settings, the date, the full response and its score, with a SHA-256 manifest of every file. Model makers who want a model added, or who think a score is wrong, can get in touch through the about page.
Frequently asked questions
How are Indic LLMs scored on this leaderboard?
How are Indic LLMs scored on this leaderboard?
Each model answers the same prompts at temperature 0. Multiple-choice, retrieval, tool-calling and routing answers are scored by code; support replies and summaries are graded 1 to 5 on language, faithfulness, task and fluency by a local LLM judge with a fixed rubric. Scores are reported from 0 to 100.
What does tokens per word measure?
What does tokens per word measure?
How many tokens a model's tokenizer needs for one word of a language, measured on the same parallel passages in every language. Dividing by the English count for the same content gives the Indic token overhead: how much more each language costs in time, context and money.
Is the judge biased toward its own answers?
Is the judge biased toward its own answers?
It may be. The judge is a local open model that is also ranked, so its own answers are flagged as self-judged in the data, and 10% of all verdicts are exported for native-speaker spot checks.
How often is the leaderboard updated?
How often is the leaderboard updated?
Monthly, and whenever a major open model is released. Every run is kept and listed in the changelog.