India · Indic LLM · Leaderboard
Which LLM should you ship in Hindi, Kannada, Tamil, Telugu, Malayalam or Hinglish?
Accuracy on the tasks a product depends on (answering from your docs, calling tools, routing support tickets, writing replies), plus what each language really costs: tokens per word, speed, memory and rupees. Every model here is open and self-hosted, run end to end by Suhas Bhairav on one machine with identical prompts.
Last run · run 2026-11-v1 · 16 models · 6 languages · run by Suhas Bhairav
Best product score per language
Hindi · हिन्दी
Gemma 4 31B
Product score 100 · 1.1× English tokens · of 14 models
Kannada · ಕನ್ನಡ
Gemma 4 31B
Product score 100 · 1.6× English tokens · of 14 models
Tamil · தமிழ்
Gemma 4 31B
Product score 100 · 1.3× English tokens · of 12 models
Telugu · తెలుగు
Gemma 4 31B
Product score 100 · 1.5× English tokens · of 12 models
Malayalam · മലയാളം
Gemma 4 31B
Product score 100 · 1.6× English tokens · of 12 models
Hinglish · Hinglish
Gemma 4 31B
Product score 100 · 1.1× English tokens · of 14 models
Full leaderboard
14 models
| # | Model | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 | Gemma 4 31B Google · 31B | 100.0 | 61.4 | 89% | 1.54 | 1.1× | 9 | 164 | 0.033 | 1.95s | 23.3 GB |
| 2 | Gemma 4 26B A4B Google · 26B | 100.0 | 58.5 | 89% | 1.54 | 1.1× | 48 | 32 | 0.006 | 0.52s | 17.5 GB |
| 3 | Gemma 4 E4B Google · 4B | 100.0 | 44.5 | 89% | 1.54 | 1.1× | 67 | 23 | 0.005 | 0.16s | 9.6 GB |
| 4 | Gemma 3 12B Google · 12B · Q4_K_M | 100.0 | 38.4 | 89% | 1.54 | 1.1× | 21 | 75 | 0.015 | 0.54s | 7.0 GB |
| 5 | Sarvam-M 24B Sarvam AI · India · 24B · Q4_K_M | 98.8 | 43.5 | 88% | 2.10 | 1.6× | 10 | 208 | 0.042 | 1.08s | 14.0 GB |
| 6 | Krutrim-2 12B Instruct Krutrim (Ola) · India · 12B · Q4_K_M | 98.8 | 32.0 | 89% | 2.10 | 1.6× | 21 | 100 | 0.020 | 0.55s | 7.0 GB |
| 7 | Aya Expanse 8B Cohere · 8B · Q4_K_M · built for hi, hinglish | 97.1 | 35.0 | 89% | 3.26 | 2.5× | 24 | 137 | 0.028 | 0.65s | 5.5 GB |
| 8 | Llama 3.1 8B Instruct Meta · 8B · Q4_K_M | 95.9 | 31.5 | 100% | 2.51 | 2.1× | 29 | 87 | 0.018 | 0.41s | 5.0 GB |
| 9 | Qwen3 8B Alibaba Qwen · 8B · Q4_K_M | 89.6 | 35.5 | 100% | 4.50 | 3.4× | 37 | 123 | 0.025 | 0.53s | 4.8 GB |
| 10 | Gemma 3 4B Google · 4B · Q4_K_M | 88.7 | 34.4 | 89% | 1.54 | 1.1× | 57 | 27 | 0.005 | 0.16s | 2.2 GB |
| 11 | Llama 3.2 3B Instruct Meta · 3B · Q4_K_M | 80.9 | 29.1 | 100% | 2.51 | 2.1× | 56 | 45 | 0.009 | 0.18s | 2.3 GB |
| 12 | Nemotron-4 Mini Hindi 4B NVIDIA · 4B · Q4_K_M · built for hi, hinglish | 61.4 | 26.1 | 67% | 1.93 | 1.4× | 53 | 36 | 0.007 | 0.17s | 1.9 GB |
| 13 | BharatGPT 3B Indic CoRover · India · 3B · Q4_K_M | 57.1 | 19.4 | 100% | 2.51 | 2.1× | 64 | 39 | 0.008 | 0.17s | 2.3 GB |
| 14 | Qwen3 30B A3B Alibaba Qwen · 30B · Q4_K_M | 27.1 | 0.0 | 0% | 4.50 | 3.4× | 58 | 78 | 0.016 | 0.37s | 18.7 GB |
Scores are 0–100. Hover a column name for its definition. Models built for specific languages appear only in those languages. The product test set is small: rank by a task to see its item count (n), and check the 95% intervals in the CSV before reading a gap of a few points as real.
Download: CSV · JSON of the results (one row per model and language, CC BY 4.0) · Datasets behind the scores
The Indic token overhead
The same message, measured with each model's own tokenizer, takes this many times the tokens it takes in English (median across models). More tokens means slower answers, shorter effective context and, on an API, a bigger bill.
Every model and language →- Hindi1.6×
- Kannada1.9×
- Tamil2.0×
- Telugu2.0×
- Malayalam2.1×
Disclosure: Kalike was built by the author
Suhas Bhairav, who runs this leaderboard, built Kalike, a small Kannada-only model (0.56B parameters). It is scored exactly like every other model: same prompts, same settings, same scorers, no tuning on this test set. Its instructions are in English like everyone else's, which a Kannada-only model handles poorly; that is reported, not corrected.
Credits and how this differs
AI4Bharat's Indic LLM-Arena ranks which answers people prefer; this leaderboard measures product tasks and running cost instead. Knowledge scores use Sarvam AI's MMLU-Indic and the IndicMMLU-Pro authors' benchmark; AI4Bharat's MILU will be added once its terms are accepted. Models come from Google, Alibaba Qwen, Meta, Cohere, OpenAI, Sarvam AI, Krutrim, NVIDIA, CoRover and CognitiveLab through Ollama. Model makers can ask for a model to be added. See the methodology.
Frequently asked questions
Which open LLM is best for Indian languages?
Which open LLM is best for Indian languages?
In the latest run (22 Nov 2026) the best product score was: Hindi: Gemma 4 31B (100); Kannada: Gemma 4 31B (100); Tamil: Gemma 4 31B (100); Telugu: Gemma 4 31B (100); Malayalam: Gemma 4 31B (100); Hinglish: Gemma 4 31B (100). Rankings change with each monthly run.
What is the Indic token overhead?
What is the Indic token overhead?
Most tokenizers split Indic scripts into many more pieces than English, so the same message costs more tokens, more time and more money. In this run the median model needed 2.1× as many tokens for Malayalam as for the same content in English.
How is this different from the AI4Bharat Indic LLM-Arena?
How is this different from the AI4Bharat Indic LLM-Arena?
The Arena ranks which answers people prefer. This leaderboard answers a product question: which model to ship in a given language, how often it gets real support, retrieval and tool-calling tasks right, and what it costs to run in tokens, time, memory and rupees.
Is Kalike scored differently because Suhas Bhairav built it?
Is Kalike scored differently because Suhas Bhairav built it?
No. Kalike, a Kannada-only model built by Suhas Bhairav, runs with exactly the same prompts, settings and scorers as every other model, and is labelled on every table.