GPT-4 vs Claude vs DeepL for Django Translations: Benchmarks, Cost, and Quality (2026)

We ran GPT-4o, Claude, and DeepL on real Django .po files across multiple languages. This article covers what we found, what the latest academic research says (WMT24 and WMT25 both had surprising results), and how to pick the right provider for your project.

TranslateBot supports all three plus over 100 additional models through LiteLLM. If you're setting up Django translations for the first time, our complete Django localization guide covers the full setup before you pick a provider.

What the research says

Before getting into our own testing, it's worth looking at the academic benchmarks. The most rigorous evaluation of machine translation happens at the annual WMT shared tasks, organized by the Association for Computational Linguistics.

WMT24: Claude wins the general translation task

The WMT24 General Machine Translation shared task evaluated translations across 11 language pairs using professional human annotators. Claude 3.5 Sonnet was the best-performing system overall, winning 9 out of 11 language pairs.

A general-purpose LLM outperformed every dedicated machine translation system submitted to the competiton. That was not expected.

LLM-based systems also outperformed professional human reference translations in several language pairs. Human references only placed in the winning quality cluster for 7 out of 11 pairs. For English-to-Ukrainian, Claude 3.5 scored 90.5 on COMET while the human reference scored 87.3.

WMT25: same story, different winner

The WMT25 evaluation expanded to 16 language pairs. Gemini 2.5 Pro came out on top, placing in the winning cluster for 14 out of 16 pairs. Human translations only made the top cluster for 6 out of 15 pairs where they were availble.

The paper notes this "highlights the inherent difficulty of translation, though it could also reflect the stylistic or lexical preferences of the annotators." Whatever the explanation, LLMs are consistently beating dedicated MT systems on these benchmarks now.

Where LLMs still fall short

The same research shows clear limitations. A study evaluating GPT-4 against human translators across 1,600 sentences found that GPT-4 performs at the level of a junior translator, producing 3.71 major errors per segment compared to 1.83 for senior translators.

Quality varies a lot by language pair. For English-to-Russian, GPT-4 nearly matches senior translators. For Chinese-to-English, it falls well behind. For low-resource language pairs, the gaps get wide: English-to-Khmer translation with an LLM scored 39.10 on XCOMET versus Google Translate's 65.88.

WMT24 also found that LLMs still struggle with specific linguistic phenomena in certain languages. Punctuation handling and future verb tenses in English-German, for instance, are areas where traditional neural MT systems actualy do better.

One more thing worth knowing: automated quality metrics like COMET can be misleading. A segment that received a high COMET score of 0.86 turned out to contain a critical misunderstanding error when reviewed by professionals. Don't rely on automated benchmarks alone.

The contenders (mid-2026)

OpenAI GPT-4o / GPT-4o-mini

OpenAI's models are the default in TranslateBot. GPT-4o-mini is the budget option with solid quality. GPT-4o handles complex translations and context-dependent text well. Both support custom instructions via TRANSLATING.md.

Anthropic Claude (Sonnet / Haiku)

Claude won WMT24's general translation task. Claude Sonnet 5 is strong at contextual understanding and tends to produce translations that read naturally rather than sounding translated. Claude Haiku 4.5 is a faster, cheaper option.

DeepL

DeepL launched a next-generation LLM-based translation model in early 2025, moving away from its traditional neural MT approach. The new model is trained on over seven years of properietary translation data. DeepL has also added custom instruction support in translation requests and expanded to over 100 languages. The free tier (500,000 characters per month) is still available.

Head-to-head comparison

Cost per million input tokens

LLM pricing is per token while DeepL charges per character, so direct comparison is tricky. For typical translation workloads, LLMs work out significantly cheaper. See the cost examples below.

Quality breakdown

European languages (German, French, Dutch, Spanish)

DeepL's new LLM model has strengthened its already-strong European language performance. GPT-4o and Claude Sonnet match it closely. The WMT24 results back this up: the quality gap between top LLM systems and dedicated MT for major European language pairs is neglible at this point.

GPT-4o-mini and Claude Haiku are a step below the premium options but handle typical Django UI strings fine. They can stumble with longer, complex sentences or specialized terminology.

Asian languages (Japanese, Chinese, Korean)

LLM-based providers still have a clear edge. GPT-4o and Claude Sonnet handle Japanese keigo (politeness levels), Chinese simplified/traditional distinctions, and Korean honorifics with more nuance. DeepL's new LLM model has narrowed the gap, but the depth of context that LLMs bring to these languages is still ahead.

The research lines up with this: GPT-4 nearly matches senior translators for English-Russian but falls significantly behind for Chinese-English. Which langauge pair you're translating matters more than which provider you pick.

Low-resource languages

If you're translating into languages like Khmer, Lao, or Burmese, current LLMs show large quality drops. Dedicated MT systems from Google still outperform LLMs by a wide margin for these languages. Most Django projects target mainstream languages, so this probably doesn't affect you, but it's worth knowing the limits.

Context and tone

LLM providers have an advantage here: they can read your TRANSLATING.md file and apply project-specific instructions to every translation. You can set terminology preferences ("translate 'cart' as 'Warenkorb', not 'Einkaufswagen'"), tone guidelines ("use informal 'du' in German"), and clarify ambigous terms.

DeepL has added a custom instructions parameter in its API, which is new. It also supports glossaries (up to 5 per request). But a TRANSLATING.md file can include product descriptions, audience notes, and style guidelines, which goes beyond what DeepL's instruction parameter can handle.

Placeholder handling

All providers handle Django's placeholder formats (%(name)s, {name}, %s, %d) reliably. TranslateBot includes explicit instructions in its prompts to preserve placeholders. In our testing, placeholder corruption was extremely rare across all providers.

Real-world cost examples

Here's what it costs to translate 500 strings (roughly 10,000 words) into 5 target languages:

GPT-4o-mini costs less than a nickel for a full medium-sized Django project translated into five languages. The premium models (GPT-4o, Claude Sonnet) stay well under a dollar. DeepL's free tier is free for small-to-medium projects within the 500,000 character monthly limit.

After the initial full translation, TranslateBot's incremental translation feature (only translating new or changed strings) keeps costs even lower. Subsequent runs typically process just a handful of strings.

Strengths and weaknesses

GPT-4o-mini

Best value for the price. Fast, solid quality across most language pairs, and nearly every team already has an OpenAI API key.

The trade-off: quality drops for complex or context-dependent text compared to GPT-4o and Claude Sonnet. Research shows GPT-4 produces overly literal translations and struggles with less common named entities.

GPT-4o

Strong translation quality across all languages we tested. Handles TRANSLATING.md context well and preserves placeholders reliably.

It costs roughly 17x more per token than GPT-4o-mini and is slower. Still affordable in absolute terms for translation workloads.

Claude Sonnet

The WMT24 winner. In our testing, Claude Sonnet produces the most natural-sounding translations, particularly when TRANSLATING.md contains detailed context. It handles formality levels (German Sie/du, Japanese keigo) better than the alternatives.

The most expensive LLM option per token. Anthropic API keys are also less common than OpenAI keys in most teams, which can add some friction when getting started.

Claude Haiku

Faster and cheaper than Sonnet, with full TRANSLATING.md support. The quality gap compared to Sonnet is noticeable for complex translations, and GPT-4o-mini often offers better value at a lower price.

DeepL

The free tier makes it the obvious choice for projects with no translation budget. European language quality is excellent, the API is fast, and the new LLM model has improved quality accross the board.

No TRANSLATING.md support, so you can't pass TranslateBot's project context. The custom instructions parameter helps but is not as flexible. Requires a separate [deepl] install extra.

Recommendation matrix

For most Django projects, start with GPT-4o-mini. It covers the widest range of use cases at the lowest cost. If quality doesn't meet your standards for specific languages, upgrade to Claude Sonnet or GPT-4o. The switch takes about 30 seconds.

How to switch providers in TranslateBot

Switching providers means changing two settings in your settings.py:

import os

# Option 1: GPT-4o-mini (default)
TRANSLATEBOT_MODEL = "gpt-4o-mini"
TRANSLATEBOT_API_KEY = os.getenv("OPENAI_API_KEY")

# Option 2: GPT-4o
TRANSLATEBOT_MODEL = "gpt-4o"
TRANSLATEBOT_API_KEY = os.getenv("OPENAI_API_KEY")

# Option 3: Claude Sonnet
TRANSLATEBOT_MODEL = "claude-sonnet-5-20250514"
TRANSLATEBOT_API_KEY = os.getenv("ANTHROPIC_API_KEY")

# Option 4: DeepL
TRANSLATEBOT_PROVIDER = "deepl"
TRANSLATEBOT_API_KEY = os.getenv("DEEPL_API_KEY")

For DeepL, you also need to install the DeepL extra:

uv add --dev translatebot-django[deepl]

No code changes, no migration, no redeployment. Change the settings, re-run the translate command, and your translations use the new provider.

Practical tips

Test before committing. Use the --dry-run flag to see what translations would look like without writing them to your .po files:

python manage.py translate --target-lang de --dry-run

Start cheap, upgrade if needed. Begin with GPT-4o-mini for your initial translation pass. Review the output. If certain languages need improvement, switch to Claude Sonnet or GPT-4o for those specific runs:

TRANSLATEBOT_MODEL=claude-sonnet-5-20250514 python manage.py translate --target-lang ja

Use TRANSLATING.md for quality gains. Before upgrading to a more expensive model, try adding a TRANSLATING.md file with terminology guidelines and tone instructions. This often improves GPT-4o-mini's output enough that you don't need the pricier model.

Mix and match. Nothing stops you from using DeepL for European languages (German, French, Spanish) and Claude Sonnet for Japanese or Korean. Run the translate command once per provider, targetting different languages each time.

Don't trust COMET scores alone. If you're evaluating translation quality with automated metrics, be aware that COMET can overestimate quality and miss critical errors. Always have a native speaker review a sample of translations for important language pairs.

Watch DeepL's free tier usage. TranslateBot's incremental translation helps, but a large initial translation run could use a significant portion of your 500,000 character monthly alowance.

Conclusion

LLMs now routinely match or beat dedicated machine translation systems on academic benchmarks. Claude won WMT24, Gemini won WMT25, and human translations are no longer guaranteed a spot in the top tier. That's a big shift from just a few years ago.

For Django projects, the practical upshot: GPT-4o-mini at $0.05 per project if you want value, Claude Sonnet or GPT-4o if you need top quality backed by peer-reviewed benchmarks, DeepL if you want the free tier. TranslateBot makes switching between them a settings change.

Once you've chosen a provider, the next step is to automate your Django translation workflow so new strings never pile up again.