Skip to content

How we measure caption accuracy

A caption file cannot show whether its words match what was said. A person has to compare them with the video. This is how we do it, and what the result can and cannot claim.

Method version 2026-12

Who measures it

A person compares the captions with the institution’s own video, by hand, clip by clip. No machine transcript is involved. CaptionPass never receives the video or audio: the institution sends a link to each sampled video, or shares a screen, and the reviewer watches it with their access. Where the platform’s player allows, it plays inside our review page; otherwise in a window of its own. Either way the video goes from the institution’s platform to the reviewer’s browser, never through our servers.

What counts as an error

Each of these counts as one error, and every kind weighs the same.

Substituted
A caption word that is not the word said. Punctuation or capitals count only when they change the meaning.
Omitted
A word said that the captions leave out. It is added to the words checked, because the captions never showed it.
Inserted
A caption word that was never said.
Speaker
A speaker label that names the wrong person, or a change of speaker the captions do not show.
Missing sound
A sound that matters to the meaning, such as laughter, a door or music, left out.

Punctuation and capitals count only when they change the meaning. Spellings that read the same, such as “colour” and “color”, are not errors.

The hard calls

Since method version 2026-11b, the reviewer follows these conventions. Each batch states the version it used.

  • Fillers are not errors, whether the captions keep them or leave them out, unless that changes the meaning. Said “um, so the, uh, cell divides”, captioned “so the cell divides”: no error.
  • False starts and repetitions are not errors when the captions tidy them, unless that changes the meaning. Said “the the mito- the mitochondria”, captioned “the mitochondria”: no error.
  • Speech with no caption counts. Every word said that no caption shows is an omitted word, even where no caption appears at all. A clip is never left out because its speech went uncaptioned. Twenty words said during a stretch with no caption on screen: twenty omitted words, added to the words checked.

Each error is marked on the word, the gap or the moment it concerns, with its time in the video, so it can be heard again. A clip is finished only when every caption line in it has been heard: played through at no more than 1.25× speed in the page, or stepped through line by line alongside the video where the page cannot follow the player. Each line records which, and the report says, for every sampled file, how many lines were played in the page and how many were stepped through by hand. When a posted video starts earlier or later than its captions, the reviewer aligns it to the captions; that offset is kept with the clip and stated on the report and the signed record.

How the words are counted

Accuracy is counted over the words said: the caption words in the clip, less any caption word that was never said (an inserted word), plus every word said that the captions leave out (an omitted word). Each error of any kind counts once, and accuracy is one less the errors divided by the words said. A word is any run of text with a letter or a digit in it. Chinese and Japanese, written without spaces, are counted one character at a time, and the report says so. Captions mostly in Thai, Lao, Khmer or Myanmar script are not drawn yet, because the method cannot yet count their words reliably. A caption line that overlaps the clip at all is checked whole.

Beside the headline we state speech accuracy (substituted, omitted and inserted words only, the figure comparable with a caption vendor’s word accuracy), the errors by kind, and how many errors the reviewer judged to change the meaning. Those are counted, not weighted: every error lowers the headline by the same amount.

How the sample is drawn

  • About 10 clips of 2 minutes, aiming for about 3,000 words a batch.
  • Each clip comes from a different file. Each next file is chosen at random from those not yet chosen, in proportion to its running time, so an hour-long lecture is twelve times as likely as a five-minute clip to be the next one. Only a library with fewer files than clips gets a second clip from one file, and the two never overlap.
  • Within a file, the clip starts at a random second. A file shorter than 2 minutes is checked whole.
  • The draw covers the whole library, the files the institution named when it asked, or one vendor’s files (below). The report says which.
  • A file that a vendor’s corrected file has replaced is no longer what viewers see, so it is not drawn, and the result counts it. A clip drawn before its file was replaced stays in the sample, marked against the captions it was drawn from.
  • Before the draw, the institution can list files it has no video for. They are set aside before the seed is used, and the result says how many files and hours that was.
  • The same seed also orders a reserve of further clips. If a drawn clip turns out to have no video, it is recorded as a non-response, listed beside the figure, and replaced by the next clip in that order while the reserve has one (a small library may have none), so nobody can choose which clips leave the sample.
  • Our server makes a random seed when the batch is drawn and keeps it with the clips and the list of files. The same seed and files always give the same clips, so the draw can be checked, and neither your institution nor our reviewer can choose the easy minutes.

What a sample can support

We publish the lower bound, not the observed rate: the accuracy we are 95% confident the captions meet or beat. It is rounded down, never up. A small sample supports less, however clean it looks.

The words come in clips, and errors bunch inside a clip: one captioner, one noisy room. So the bound treats the clips, not the words, as the sample (Korn and Graubard’s method: a one-sided Clopper-Pearson bound on the words divided by the design effect of the clips, with a t correction for the number of clips). When errors are spread out it is close to a bound on the words alone; when they bunch it is lower, as it should be. A batch with one reviewed clip states its counts and no bound.

It describes the captioned minutes of the files the sample was drawn from, on the day it was drawn, and the report names them.

Sample sizes and the accuracy each supports with 95% confidence
Words checkedErrors foundObservedSupports at least
3000100.0%99.0%
800399.6%99.0%
3,0001099.6%99.4%

Ten clips of 300 words each, with the same 10 errors placed differently:

Where the errors fall, and the bound with clips as the sample
Where the 10 errors fallBound on the words aloneWith clips as the sample
One in each clip99.4%99.3%
Five each in two clips99.4%98.8%
All ten in one clip99.4%98.0%

About 3,000 words can show at least 99.0% only with 20 errors or fewer (observed 99.33% or better); errors bunched in a few clips lower what it shows. Captions that run close to 99% cannot be shown to meet 99% with one batch.

The library report shows the bound for the whole batch, with its files, words and errors. A file page shows that file’s sampled minutes, its words and errors, and the batch’s bound: a clip or two cannot support a bound for a whole file. Batches recorded before method version 2026-12 keep the figures they were recorded with, labelled with their version.

Accuracy by vendor

If your captions come from more than one vendor, you can ask for the sample to be drawn from one vendor’s files. It is a whole sample of its own: about 10 clips, its own seed, drawn from the files your team assigned to that vendor, then reviewed, recorded and signed like any other, with the record naming the vendor as your team assigned it. Its figure is shown under the vendor’s name and never as your library’s accuracy. Measuring a further vendor is another sample, which we quote.

We never split one library sample by vendor. Ten clips shared by three vendors leaves three or four each, and that few clips cannot tell vendors apart. With 300 words said in each clip, one error per clip is 99.7% accurate and three per clip is 99.0%. We say one vendor is more accurate only when its lower bound is above the other’s upper bound:

Clips per vendor, and whether two vendors’ bounds separate
Clips per vendorAt least, with 1 error a clipAt most, with 3 errors a clipTells them apart
398.34%99.72%No
599.13%99.48%No
1099.35%99.28%Yes
2099.48%99.20%Yes

Errors bunched in one clip widen both bounds further. When we compare how often each vendor’s files fail a check, we count your own files exactly, and we say a vendor fails a check more or less often than the rest only for a vendor with at least 20 files, and only when Fisher’s exact test finds the difference at 5% after Holm’s correction across every comparison on the page. For one comparison, telling 10% of files failing from 30% takes 40 files per vendor, and 10% from 20% takes 113.

No video, no claim

Without the video for a file, its accuracy is not measured, and its page and the report say so. A clip without video is left out of the batch, never counted as a pass. A file that was never sampled says “not measured” too. Nothing we publish implies an accuracy that a person did not measure.

A clip is left out only for a recorded reason: the institution said after the draw that there is no video for its file (a non-response, replaced from the reserve), or the file left the library before the clip was reviewed. A file in a sample being reviewed cannot be deleted until the sample is recorded. The report lists every clip left out, and why.

Once recorded, it is fixed

When every clip with a video is reviewed, the batch is recorded. Its result then shows on the files and the report, with the list of every error found: the clip, the time, the kind, the caption word and what was said, so you can check each one against your own video. Later edits cannot change it. A new measurement is a new batch.

A signed record you can check

Recording a batch issues a signed accuracy record: the files the sample was drawn from, each with its fingerprint; the seed; every clip, with the video used and how it ended; every error; and the figure. It is signed the same way as the record for each caption file, so anyone holding a copy can confirm on the verify page that it is what CaptionPass issued and that nothing in it has changed. It never holds an email address or the reviewer’s time.

You do not have to trust our arithmetic either. Paste a record below and this page redraws its clips from its seed and files, and recomputes its counts and figure, with the same code we run. It runs in your browser and sends nothing to us.

Versions

2026-11
The first published method: five error kinds, words checked, the seeded draw and the one-sided 95% bound.
2026-11b
Writes down the marking conventions: fillers, false starts and repetitions are not errors unless they change the meaning, and speech with no caption counts as omitted words.
2026-12
The bound treats clips, not words, as the sampling unit (Korn-Graubard), and names the files it describes. Accuracy is counted over the words said (inserted caption words were never said, so they leave the denominator), with speech accuracy, errors by kind and errors that change the meaning stated beside it. Files without video are set aside before the draw; a drawn clip later found to have no video is replaced from the same seeded order and listed. Chinese and Japanese are counted in characters, and files mostly in Thai, Lao, Khmer or Myanmar script are not drawn yet.

Questions about a result? Ask from your dashboard, or read what a CaptionPass record proves.