India · Indic LLM · Leaderboard

Which LLM should you ship in Hindi, Kannada, Tamil, Telugu, Malayalam or Hinglish?

Accuracy on the tasks a product depends on (answering from your docs, calling tools, routing support tickets, writing replies), plus what each language really costs: tokens per word, speed, memory and rupees. Every model here is open and self-hosted, run end to end by Suhas Bhairav on one machine with identical prompts.

Last run · run 2026-11-v1 · 16 models · 6 languages · run by Suhas Bhairav

Best product score per language

  • Hindi · हिन्दी

    Gemma 4 31B

    Product score 100 · 1.1× English tokens · of 14 models

  • Kannada · ಕನ್ನಡ

    Gemma 4 31B

    Product score 100 · 1.6× English tokens · of 14 models

  • Tamil · தமிழ்

    Gemma 4 31B

    Product score 100 · 1.3× English tokens · of 12 models

  • Telugu · తెలుగు

    Gemma 4 31B

    Product score 100 · 1.5× English tokens · of 12 models

  • Malayalam · മലയാളം

    Gemma 4 31B

    Product score 100 · 1.6× English tokens · of 12 models

  • Hinglish · Hinglish

    Gemma 4 31B

    Product score 100 · 1.1× English tokens · of 14 models

Full leaderboard

Built in

14 models

#Model
1
Gemma 4 31B
Google · 31B
100.061.489%1.541.1×91640.0331.95s23.3 GB
2
Gemma 4 26B A4B
Google · 26B
100.058.589%1.541.1×48320.0060.52s17.5 GB
3
Gemma 4 E4B
Google · 4B
100.044.589%1.541.1×67230.0050.16s9.6 GB
4
Gemma 3 12B
Google · 12B · Q4_K_M
100.038.489%1.541.1×21750.0150.54s7.0 GB
5
Sarvam-M 24B
Sarvam AI · India · 24B · Q4_K_M
98.843.588%2.101.6×102080.0421.08s14.0 GB
6
Krutrim-2 12B Instruct
Krutrim (Ola) · India · 12B · Q4_K_M
98.832.089%2.101.6×211000.0200.55s7.0 GB
7
Aya Expanse 8B
Cohere · 8B · Q4_K_M · built for hi, hinglish
97.135.089%3.262.5×241370.0280.65s5.5 GB
8
Llama 3.1 8B Instruct
Meta · 8B · Q4_K_M
95.931.5100%2.512.1×29870.0180.41s5.0 GB
9
Qwen3 8B
Alibaba Qwen · 8B · Q4_K_M
89.635.5100%4.503.4×371230.0250.53s4.8 GB
10
Gemma 3 4B
Google · 4B · Q4_K_M
88.734.489%1.541.1×57270.0050.16s2.2 GB
11
Llama 3.2 3B Instruct
Meta · 3B · Q4_K_M
80.929.1100%2.512.1×56450.0090.18s2.3 GB
12
Nemotron-4 Mini Hindi 4B
NVIDIA · 4B · Q4_K_M · built for hi, hinglish
61.426.167%1.931.4×53360.0070.17s1.9 GB
13
BharatGPT 3B Indic
CoRover · India · 3B · Q4_K_M
57.119.4100%2.512.1×64390.0080.17s2.3 GB
14
Qwen3 30B A3B
Alibaba Qwen · 30B · Q4_K_M
27.10.00%4.503.4×58780.0160.37s18.7 GB

Scores are 0–100. Hover a column name for its definition. Models built for specific languages appear only in those languages. The product test set is small: rank by a task to see its item count (n), and check the 95% intervals in the CSV before reading a gap of a few points as real.

Download: CSV · JSON of the results (one row per model and language, CC BY 4.0) · Datasets behind the scores

The Indic token overhead

The same message, measured with each model's own tokenizer, takes this many times the tokens it takes in English (median across models). More tokens means slower answers, shorter effective context and, on an API, a bigger bill.

Every model and language →
  • Hindi1.6×
  • Kannada1.9×
  • Tamil2.0×
  • Telugu2.0×
  • Malayalam2.1×

Disclosure: Kalike was built by the author

Suhas Bhairav, who runs this leaderboard, built Kalike, a small Kannada-only model (0.56B parameters). It is scored exactly like every other model: same prompts, same settings, same scorers, no tuning on this test set. Its instructions are in English like everyone else's, which a Kannada-only model handles poorly; that is reported, not corrected.

Credits and how this differs

AI4Bharat's Indic LLM-Arena ranks which answers people prefer; this leaderboard measures product tasks and running cost instead. Knowledge scores use Sarvam AI's MMLU-Indic and the IndicMMLU-Pro authors' benchmark; AI4Bharat's MILU will be added once its terms are accepted. Models come from Google, Alibaba Qwen, Meta, Cohere, OpenAI, Sarvam AI, Krutrim, NVIDIA, CoRover and CognitiveLab through Ollama. Model makers can ask for a model to be added. See the methodology.

Frequently asked questions

Which open LLM is best for Indian languages?

In the latest run (22 Nov 2026) the best product score was: Hindi: Gemma 4 31B (100); Kannada: Gemma 4 31B (100); Tamil: Gemma 4 31B (100); Telugu: Gemma 4 31B (100); Malayalam: Gemma 4 31B (100); Hinglish: Gemma 4 31B (100). Rankings change with each monthly run.

What is the Indic token overhead?

Most tokenizers split Indic scripts into many more pieces than English, so the same message costs more tokens, more time and more money. In this run the median model needed 2.1× as many tokens for Malayalam as for the same content in English.

How is this different from the AI4Bharat Indic LLM-Arena?

The Arena ranks which answers people prefer. This leaderboard answers a product question: which model to ship in a given language, how often it gets real support, retrieval and tool-calling tasks right, and what it costs to run in tokens, time, memory and rupees.

Is Kalike scored differently because Suhas Bhairav built it?

No. Kalike, a Kannada-only model built by Suhas Bhairav, runs with exactly the same prompts, settings and scorers as every other model, and is labelled on every table.