Symbols per token in different LLM models

Symbols per token in different LLM models for different languages

Language / ModelGPT-4o (2024-08-06)GPT-4o-mini (2024-07-18)GPT-4Claude-3-5-Sonnet (2024-06-20)
EnglishSymbols: 4.63Symbols: 4.60Symbols: 4.77Symbols: 4.41
Words: 0.77Words: 0.77Words: 0.79Words: 0.74
FrenchSymbols: 3.65Symbols: 3.65Symbols: 3.29Symbols: 2.81
Words: 0.71Words: 0.71Words: 0.64Words: 0.55
RomanianSymbols: 3.75Symbols: 3.75Symbols: 3.36Symbols: 3.34
Words: 0.59Words: 0.59Words: 0.53Words: 0.52
RussianSymbols: 3.93Symbols: 3.90Symbols: 2.57Symbols: 2.74
Words: 0.55Words: 0.55Words: 0.36Words: 0.39
UkrainianSymbols: 2.59Symbols: 2.59Symbols: 1.64Symbols: 2.12
Words: 0.45Words: 0.45Words: 0.28Words: 0.37

While preparing a presentation for the employees, I had to analyze the current “expressiveness” of tokens for different languages. The conclusions are as follows:

  • Responses in some languages cost three times more than in English.
  • Comparing models by price per token is only possible for the English language. For other languages, the cost needs to be recalculated into the price per character, word, or sentence.
  • In some cases, it might be more cost-effective to translate the request and response into English. However, this needs to be carefully calculated, especially for the responses.
  • It is difficult to predict which language is “more expensive” and which is “cheaper.” Calculations are necessary.

These conclusions will help us better understand and optimize the costs of using different languages in our models.

  1. Token efficiency and linguistic structure: English shows that LLMs process it more efficiently than Russian or Ukrainian.
  2. Impact of model design on multilingual processing: Claude-3-5-Sonnet may tokenize text into smaller units.
  3. Consequences for cost and speed: Lower symbol-to-token ratios lead to higher processing costs for Cyrillic languages.
  4. Alphabet and tokenization mechanisms: Cyrillic symbols are handled less efficiently by current tokenizers.
  5. Morphological complexity vs. tokenization: Simpler structures tokenize more efficiently.