Efficiency
The Indic token overhead
The same passages and messages, written in each language, counted with each model's own tokenizer. A value of 5× means a Kannada message costs five times the tokens of the same message in English: five times the API bill, a fifth of the effective context window, and much slower answers on the same hardware.
Last run · run 2026-11-v1 · 16 models · 6 languages · run by Suhas Bhairav
| Model | English | Hindi | Kannada | Tamil | Telugu | Malayalam | Hinglish |
|---|---|---|---|---|---|---|---|
| Kalike d20Suhas Bhairav | 1.0×5.5/word | – | 0.3×1.9/word | – | – | – | – |
| Ambari 7B Instruct v0.1CognitiveLab | 1.0×1.6/word | – | 1.1×2.3/word | – | – | – | – |
| Nemotron-4 Mini Hindi 4BNVIDIA | 1.0×1.5/word | 1.4×1.9/word | – | – | – | – | 1.2×1.7/word |
| Gemma 4 31BGoogle | 1.0×1.5/word | 1.1×1.5/word | 1.6×3.1/word | 1.3×2.5/word | 1.5×2.9/word | 1.6×3.1/word | 1.1×1.6/word |
| Gemma 4 26B A4BGoogle | 1.0×1.5/word | 1.1×1.5/word | 1.6×3.1/word | 1.3×2.5/word | 1.5×2.9/word | 1.6×3.1/word | 1.1×1.6/word |
| Gemma 4 E4BGoogle | 1.0×1.5/word | 1.1×1.5/word | 1.6×3.1/word | 1.3×2.5/word | 1.5×2.9/word | 1.6×3.1/word | 1.1×1.6/word |
| Gemma 3 12BGoogle | 1.0×1.5/word | 1.1×1.5/word | 1.6×3.1/word | 1.3×2.5/word | 1.5×2.9/word | 1.6×3.1/word | 1.1×1.6/word |
| Gemma 3 4BGoogle | 1.0×1.5/word | 1.1×1.5/word | 1.6×3.1/word | 1.3×2.5/word | 1.5×2.9/word | 1.6×3.1/word | 1.1×1.6/word |
| Sarvam-M 24BSarvam AI | 1.0×1.5/word | 1.6×2.1/word | 1.9×3.7/word | 2.0×3.7/word | 2.0×3.6/word | 2.1×4.1/word | 1.2×1.7/word |
| Krutrim-2 12B InstructKrutrim (Ola) | 1.0×1.5/word | 1.6×2.1/word | 1.9×3.7/word | 2.0×3.7/word | 2.0×3.6/word | 2.1×4.1/word | 1.2×1.7/word |
| Aya Expanse 8BCohere | 1.0×1.5/word | 2.5×3.3/word | – | – | – | – | 1.2×1.7/word |
| Qwen3 8BAlibaba Qwen | 1.0×1.5/word | 3.4×4.5/word | 5.3×10.2/word | 4.8×8.9/word | 5.6×10.4/word | 5.4×10.5/word | 1.3×1.8/word |
| Qwen3 30B A3BAlibaba Qwen | 1.0×1.5/word | 3.4×4.5/word | 5.3×10.2/word | 4.8×8.9/word | 5.6×10.4/word | 5.4×10.5/word | 1.3×1.8/word |
| Llama 3.1 8B InstructMeta | 1.0×1.4/word | 2.1×2.5/word | 7.0×12.4/word | 6.3×10.7/word | 6.8×11.8/word | 6.9×12.4/word | 1.3×1.6/word |
| Llama 3.2 3B InstructMeta | 1.0×1.4/word | 2.1×2.5/word | 7.0×12.4/word | 6.3×10.7/word | 6.8×11.8/word | 6.9×12.4/word | 1.3×1.6/word |
| BharatGPT 3B IndicCoRover | 1.0×1.4/word | 2.1×2.5/word | 7.0×12.4/word | 6.3×10.7/word | 6.8×11.8/word | 6.9×12.4/word | 1.3×1.6/word |
Each cell: tokens for the language's version of the same text divided by tokens for the English version, and tokens per whitespace word. Words are not comparable across languages (Dravidian languages pack more into one word), which is why the multiplier against English, on identical content, is the number to budget with. Kalike's tokenizer is Kannada-only, so it scores far below 1× on Kannada and very high on English.
Frequently asked questions
Why do Indian languages use more tokens than English?
Why do Indian languages use more tokens than English?
Most tokenizers are trained mostly on English and Latin-script text, so they learn few whole Indic words and split Devanagari, Kannada, Tamil, Telugu and Malayalam text into many small pieces. The same sentence then needs several times the tokens.
How many more tokens does each Indian language need?
How many more tokens does each Indian language need?
Median across the models in the latest run, for the same content as English: Hindi 1.6×, Kannada 1.9×, Tamil 2.0×, Telugu 2.0×, Malayalam 2.1×.
Which tokenizer is most efficient for each language?
Which tokenizer is most efficient for each language?
Hindi: Gemma 4 31B (1.1×); Kannada: Kalike d20 (0.3×); Tamil: Gemma 4 31B (1.3×); Telugu: Gemma 4 31B (1.5×); Malayalam: Gemma 4 31B (1.6×).