Indic LLM · Datasets

The datasets behind the scores

The product score comes from a private set of real product tasks in Hindi, Kannada, Tamil, Telugu, Malayalam and Hinglish. The knowledge score comes from two public benchmarks, Sarvam AI's MMLU-Indic and IndicMMLU-Pro, with AI4Bharat's MILU planned. Public datasets are linked to their source and not copied here.

Last run · run 2026-11-v1 · 16 models · 6 languages · run by Suhas Bhairav

Indic product tasks (private set)

In use · private

By Suhas Bhairav

What it tests
Short, realistic tasks an Indian product actually handles: answer a customer from a policy or product passage (and say NOT_FOUND when the passage has no answer), turn a request into a JSON tool call, route a support message, write a support reply, and summarise local news.
Languages
Hindi, Kannada, Tamil, Telugu, Malayalam, Hinglish
How it is used
Every scenario is parallel across the six languages: the same facts and the same correct answer, so scores and token counts compare like for like. Companies, prices and people are fictional. Instructions are in English; the user-facing text is in the target language.
Scoring
Retrieval Q&A, tool calling and routing are scored by code (exact facts, valid JSON with correct arguments, one correct category). The support reply and summary are graded by a local LLM judge with a fixed four-part rubric.
Access
Kept private so models cannot be trained on it, which keeps the rankings meaningful. Version 0 was drafted with an AI assistant and is pending native-speaker review; the next version targets 200–500 items per language, written or checked by native speakers.

MMLU-Indic

In use

By Sarvam AI · licence MIT

What it tests
Sarvam AI's translation of MMLU, the widely used general-knowledge benchmark (57 subjects, four options per question), into Indic languages, with romanised versions as well.
Languages
Hindi, Kannada, Tamil, Telugu, Malayalam, romanised Hindi (used for Hinglish)
How it is used
A seeded sample of 100 test questions per language. The rows are parallel across languages, so every language is asked the same questions. The romanised Hindi set stands in for Hinglish knowledge.
Scoring
Multiple choice, exact match on the option letter.
Access
Public on Hugging Face under the MIT licence.
sarvamai/mmlu-indic on Hugging Face ↗

IndicMMLU-Pro

In use

By IndicMMLU-Pro authors (LinguaLift)

What it tests
An Indic extension of MMLU-Pro: harder, reasoning-heavy questions with up to ten options, translated into nine Indic languages. It has no Malayalam set.
Languages
Hindi, Kannada, Tamil, Telugu
How it is used
A seeded sample of 60 test questions per language, stratified by source subject. Questions that became only blanks during translation (about 0.5%) are dropped.
Scoring
Multiple choice, exact match on the option letter (A–J).
Access
Public on Hugging Face. The dataset card does not state a licence; check it before reusing the questions.
LinguaLift/IndicMMLU-Pro on Hugging Face ↗

MILU (Multi-task Indic Language Understanding)

Planned

By AI4Bharat

What it tests
AI4Bharat's benchmark of India-specific knowledge: 8 domains and 41 subjects across 11 Indic languages, including regional history, law, governance and culture that translated Western benchmarks miss.
Languages
Hindi, Kannada, Tamil, Telugu, Malayalam, and six more
How it is used
Planned at 100 test questions per language, stratified by subject, once its terms are accepted.
Scoring
Multiple choice, exact match.
Access
Gated on Hugging Face: access requires accepting AI4Bharat's terms. It is not yet in the scores.
ai4bharat/MILU on Hugging Face ↗

How the scores are combined is on the methodology page. The leaderboard's own results (not the questions) can be downloaded as CSV or JSON.

Frequently asked questions

Which datasets does the Indic LLM Leaderboard use?

A private set of product tasks (retrieval Q&A, tool calling, support routing, support replies, summaries) in Hindi, Kannada, Tamil, Telugu, Malayalam and Hinglish, plus two public knowledge benchmarks: Sarvam AI's MMLU-Indic and IndicMMLU-Pro. AI4Bharat's MILU is planned.

Why is the product test set private?

Public test questions end up in training data, and a model that has seen the test scores well without being better. Keeping the product set private keeps the rankings honest. Public benchmarks are used for the knowledge score and linked here.

Can I download the benchmark questions from this site?

No. The public benchmarks are linked to their original pages on Hugging Face, where their creators publish them under their own terms. This site publishes only the leaderboard results, as CSV and JSON.

Is there an Indic benchmark for Hinglish?

Few exist. The leaderboard writes its own Hinglish (Hindi-English mix in Latin script) product tasks and uses Sarvam AI's romanised Hindi MMLU for Hinglish knowledge.