Efficiency

The Indic token overhead

The same passages and messages, written in each language, counted with each model's own tokenizer. A value of 5× means a Kannada message costs five times the tokens of the same message in English: five times the API bill, a fifth of the effective context window, and much slower answers on the same hardware.

Last run · run 2026-11-v1 · 16 models · 6 languages · run by Suhas Bhairav

Tokens vs English, same content1×7.0×
Tokens needed relative to English for the same content, and tokens per word, by model and language
ModelEnglishHindiKannadaTamilTeluguMalayalamHinglish
Kalike d20Suhas Bhairav1.0×5.5/word–0.3×1.9/word––––
Ambari 7B Instruct v0.1CognitiveLab1.0×1.6/word–1.1×2.3/word––––
Nemotron-4 Mini Hindi 4BNVIDIA1.0×1.5/word1.4×1.9/word––––1.2×1.7/word
Gemma 4 31BGoogle1.0×1.5/word1.1×1.5/word1.6×3.1/word1.3×2.5/word1.5×2.9/word1.6×3.1/word1.1×1.6/word
Gemma 4 26B A4BGoogle1.0×1.5/word1.1×1.5/word1.6×3.1/word1.3×2.5/word1.5×2.9/word1.6×3.1/word1.1×1.6/word
Gemma 4 E4BGoogle1.0×1.5/word1.1×1.5/word1.6×3.1/word1.3×2.5/word1.5×2.9/word1.6×3.1/word1.1×1.6/word
Gemma 3 12BGoogle1.0×1.5/word1.1×1.5/word1.6×3.1/word1.3×2.5/word1.5×2.9/word1.6×3.1/word1.1×1.6/word
Gemma 3 4BGoogle1.0×1.5/word1.1×1.5/word1.6×3.1/word1.3×2.5/word1.5×2.9/word1.6×3.1/word1.1×1.6/word
Sarvam-M 24BSarvam AI1.0×1.5/word1.6×2.1/word1.9×3.7/word2.0×3.7/word2.0×3.6/word2.1×4.1/word1.2×1.7/word
Krutrim-2 12B InstructKrutrim (Ola)1.0×1.5/word1.6×2.1/word1.9×3.7/word2.0×3.7/word2.0×3.6/word2.1×4.1/word1.2×1.7/word
Aya Expanse 8BCohere1.0×1.5/word2.5×3.3/word––––1.2×1.7/word
Qwen3 8BAlibaba Qwen1.0×1.5/word3.4×4.5/word5.3×10.2/word4.8×8.9/word5.6×10.4/word5.4×10.5/word1.3×1.8/word
Qwen3 30B A3BAlibaba Qwen1.0×1.5/word3.4×4.5/word5.3×10.2/word4.8×8.9/word5.6×10.4/word5.4×10.5/word1.3×1.8/word
Llama 3.1 8B InstructMeta1.0×1.4/word2.1×2.5/word7.0×12.4/word6.3×10.7/word6.8×11.8/word6.9×12.4/word1.3×1.6/word
Llama 3.2 3B InstructMeta1.0×1.4/word2.1×2.5/word7.0×12.4/word6.3×10.7/word6.8×11.8/word6.9×12.4/word1.3×1.6/word
BharatGPT 3B IndicCoRover1.0×1.4/word2.1×2.5/word7.0×12.4/word6.3×10.7/word6.8×11.8/word6.9×12.4/word1.3×1.6/word

Each cell: tokens for the language's version of the same text divided by tokens for the English version, and tokens per whitespace word. Words are not comparable across languages (Dravidian languages pack more into one word), which is why the multiplier against English, on identical content, is the number to budget with. Kalike's tokenizer is Kannada-only, so it scores far below 1× on Kannada and very high on English.

Frequently asked questions

Why do Indian languages use more tokens than English?

Most tokenizers are trained mostly on English and Latin-script text, so they learn few whole Indic words and split Devanagari, Kannada, Tamil, Telugu and Malayalam text into many small pieces. The same sentence then needs several times the tokens.

How many more tokens does each Indian language need?

Median across the models in the latest run, for the same content as English: Hindi 1.6×, Kannada 1.9×, Tamil 2.0×, Telugu 2.0×, Malayalam 2.1×.

Which tokenizer is most efficient for each language?

Hindi: Gemma 4 31B (1.1×); Kannada: Kalike d20 (0.3×); Tamil: Gemma 4 31B (1.3×); Telugu: Gemma 4 31B (1.5×); Malayalam: Gemma 4 31B (1.6×).