Indic LLM · Datasets
The datasets behind the scores
The product score comes from a private set of real product tasks in Hindi, Kannada, Tamil, Telugu, Malayalam and Hinglish. The knowledge score comes from two public benchmarks, Sarvam AI's MMLU-Indic and IndicMMLU-Pro, with AI4Bharat's MILU planned. Public datasets are linked to their source and not copied here.
Last run · run 2026-11-v1 · 16 models · 6 languages · run by Suhas Bhairav
Indic product tasks (private set)
In use · privateBy Suhas Bhairav
- What it tests
- Short, realistic tasks an Indian product actually handles: answer a customer from a policy or product passage (and say NOT_FOUND when the passage has no answer), turn a request into a JSON tool call, route a support message, write a support reply, and summarise local news.
- Languages
- Hindi, Kannada, Tamil, Telugu, Malayalam, Hinglish
- How it is used
- Every scenario is parallel across the six languages: the same facts and the same correct answer, so scores and token counts compare like for like. Companies, prices and people are fictional. Instructions are in English; the user-facing text is in the target language.
- Scoring
- Retrieval Q&A, tool calling and routing are scored by code (exact facts, valid JSON with correct arguments, one correct category). The support reply and summary are graded by a local LLM judge with a fixed four-part rubric.
- Access
- Kept private so models cannot be trained on it, which keeps the rankings meaningful. Version 0 was drafted with an AI assistant and is pending native-speaker review; the next version targets 200–500 items per language, written or checked by native speakers.
MMLU-Indic
In use- What it tests
- Sarvam AI's translation of MMLU, the widely used general-knowledge benchmark (57 subjects, four options per question), into Indic languages, with romanised versions as well.
- Languages
- Hindi, Kannada, Tamil, Telugu, Malayalam, romanised Hindi (used for Hinglish)
- How it is used
- A seeded sample of 100 test questions per language. The rows are parallel across languages, so every language is asked the same questions. The romanised Hindi set stands in for Hinglish knowledge.
- Scoring
- Multiple choice, exact match on the option letter.
- Access
- Public on Hugging Face under the MIT licence.
IndicMMLU-Pro
In useBy IndicMMLU-Pro authors (LinguaLift)
- What it tests
- An Indic extension of MMLU-Pro: harder, reasoning-heavy questions with up to ten options, translated into nine Indic languages. It has no Malayalam set.
- Languages
- Hindi, Kannada, Tamil, Telugu
- How it is used
- A seeded sample of 60 test questions per language, stratified by source subject. Questions that became only blanks during translation (about 0.5%) are dropped.
- Scoring
- Multiple choice, exact match on the option letter (A–J).
- Access
- Public on Hugging Face. The dataset card does not state a licence; check it before reusing the questions.
MILU (Multi-task Indic Language Understanding)
PlannedBy AI4Bharat
- What it tests
- AI4Bharat's benchmark of India-specific knowledge: 8 domains and 41 subjects across 11 Indic languages, including regional history, law, governance and culture that translated Western benchmarks miss.
- Languages
- Hindi, Kannada, Tamil, Telugu, Malayalam, and six more
- How it is used
- Planned at 100 test questions per language, stratified by subject, once its terms are accepted.
- Scoring
- Multiple choice, exact match.
- Access
- Gated on Hugging Face: access requires accepting AI4Bharat's terms. It is not yet in the scores.
How the scores are combined is on the methodology page. The leaderboard's own results (not the questions) can be downloaded as CSV or JSON.
Frequently asked questions
Which datasets does the Indic LLM Leaderboard use?
Which datasets does the Indic LLM Leaderboard use?
A private set of product tasks (retrieval Q&A, tool calling, support routing, support replies, summaries) in Hindi, Kannada, Tamil, Telugu, Malayalam and Hinglish, plus two public knowledge benchmarks: Sarvam AI's MMLU-Indic and IndicMMLU-Pro. AI4Bharat's MILU is planned.
Why is the product test set private?
Why is the product test set private?
Public test questions end up in training data, and a model that has seen the test scores well without being better. Keeping the product set private keeps the rankings honest. Public benchmarks are used for the knowledge score and linked here.
Can I download the benchmark questions from this site?
Can I download the benchmark questions from this site?
No. The public benchmarks are linked to their original pages on Hugging Face, where their creators publish them under their own terms. This site publishes only the leaderboard results, as CSV and JSON.
Is there an Indic benchmark for Hinglish?
Is there an Indic benchmark for Hinglish?
Few exist. The leaderboard writes its own Hinglish (Hindi-English mix in Latin script) product tasks and uses Sarvam AI's romanised Hindi MMLU for Hinglish knowledge.