Equal CER, Unequal Damage

Severity-Weighted Character Error Rate for OCR

Nguyen Nguyen Yongyi Zang

CER charges a typo as much as a wrong name.
semCER charges each error what it costs a reader.

Break it yourself

Make an OCR error. CER charges every one the same. semCER charges what it costs a reader.

Try one:

Original · tap any letter to change it

Take 2 tablets at 8:30, not after meals.

What OCR read

Take 2 tablets at 8:30, not after meals.

CER

1 error

Pick an error to see the scores.

semCER

–

 

See real OCR errors →

Abstract

Character Error Rate (CER), the standard metric for OCR and vision-language models (VLMs), charges a typo as much as a wrong name. We introduce semCER, which keeps CER's alignment and denominator and charges each grapheme edit a severity from an LLM judge, with no word segmentation. Normalized CER keeps the score of a whole transcription, where it tracks readers as well as semCER or better, and semCER breaks that score down into named edits. The charges track readers' ratings edit by edit in English, Vietnamese, and Chinese and put the edits readers rate most severe near the top of a sorted list. On pairs drawn where CER and semCER disagreed, readers side with semCER more often than with normalized CER in English and Chinese, and equally often in Vietnamese. Across 16 languages, the judge charges a wrong diacritic or tone mark almost nothing in Spanish but nearly a wrong letter's severity in Telugu. Where we can count real words, a general VLM's wrong letter forms one far above chance while a classical pipeline's stays near it. We release the 61 million judged edits, the aggregated reader ratings, and the distilled scorer.

How it works

Same alignment, new price tag

semCER finds errors exactly the way CER does. It only changes what each one costs: an LLM judge reads the whole line and rates each error from 0 to 1, and a 0.8B model distilled from it does the same on your laptop. Flip between the two price lists.

semCER 5.4%1.90 ÷ 35 characters

    What we found

    What the price tag reveals

    1

    Short on time? Fix the expensive errors first

    Sort a pile of OCR errors by semCER, and the first 10% you check catch 81 to 92% of the damage that readers’ own top picks would. Sort by CER, and you might as well pick at random.

    Drag across the chart to see which real errors each order checks.

    Check errors in order. How much of the damage have you found?

    Lines with one error each, rated by readers; damage is the sum of their ratings

    2

    Where the two scores disagree, readers mostly side with semCER

    We gave readers two transcriptions of the same line that CER and semCER ranked in opposite order, and let them pick the better one.

    In English and Chinese, semCER matched their pick more often than CER did, even after CER ignores case, spacing and punctuation. In Vietnamese, the two tied.

    How often each score picked the transcription readers preferred

    3

    An accent is cheap in Spanish and costly in Telugu

    In Spanish, a dropped accent like más → mas costs almost nothing. In Telugu or Kannada, a wrong mark costs almost as much as a wrong letter.

    CER charges 1 either way. semCER follows what the mark does in each language.

    How bad is a wrong accent, compared with a wrong letter?

    Mean cost of each kind of error per language, from the LLM judge

    Why the spread? It follows what the marks do more than the type of script. A Spanish or French accent mostly decorates a letter. Vietnamese tone marks and the vowel signs of Hindi, Thai and Telugu change the word, so Vietnamese, an alphabet, sits next to Hindi.

    4

    Vision-language models make wrong words, not just typos

    A classic OCR engine’s wrong letter usually makes a non-word that a spell checker would flag. A vision-language model’s wrong letter makes a real word far more often, one that reads fine and slips through. semCER charges its letter errors more.

    AVLMs turn far more of their wrong letters into real words

    Share of wrong letters that form a real word

    BTheir letter errors cost more, everywhere but Spanish

    Mean cost of letter errors, 0 to 1, from the LLM judge

    classic OCRgeneral VLM

    Use it

    Runs on your laptop

    0.8Bparameter open model
    16languages
    9 in 10clear pairs of errors ranked as the LLM judge ranks them

    Code

    pip install semcer
    import semcer

result = semcer.score(
    reference="The total is 15 dollars.",
    hypothesis="The total is 75 dollars.",
)
print(result.explain())

    Output

    semcer=0.0414  CER=0.0417  (1 edit(s) over 24 reference graphemes; lower is better)
  SUB ref[13:14] '1' -> '7'  severity 0.99 (meaning lost)

    One wrong digit. CER sees one error in 24 characters. semCER sees that the price changed.

    BibTeX

    @misc{nguyen2026equalcer,
  title={Equal CER, Unequal Damage: Severity-Weighted Character Error Rate for OCR},
  author={Nguyen Nguyen and Yongyi Zang},
  year={2026},
  eprint={2610.20461},
  archivePrefix={arXiv},
  primaryClass={cs.CL},
  url={https://arxiv.org/abs/2610.20461},
}