Word Wizards toolbox with different tools.

CAIV Methodology

CAIV Methodology and Process

CAIV studies are based on a modified WER (Word Error Rate) methodology which provides an objective percentage of word accuracy based on total word errors and omissions, taking context, formatting, and punctuation out of the accuracy calculation. This makes for a more empirical data-driven methodology compared to the other methods of speech-to-text analysis. To eliminate the subjectivity of punctuation and speaker identification inherent in AI platforms and live human captioning, CAIV can provide a separate score for speaker identification and/or punctuation accuracy, viewable alongside our WER based percentage score.

WER vs FER vs NER

The FER method (Formatted Error Rate) is based on word errors when formatting elements such as punctuation, grammar, speaker identification, non-speech elements, capitalization, and other notations are included. Because of the subjectivity of these elements and the inconsistency between AI and live human captioning, we recommend WER over FER for true accuracy reporting and provide a separate score for punctuation and speaker change performance.

The NER method, (Number, Edition, and Recognition Error) provides a subjective “score” out of 100 rather than a true percentage of accuracy. NER scoring seeks to convey how accurately ideas (but not individual words) are captured in captions, making it a subjective and potentially legally risky measurement. It also includes punctuation, which adds variability to the study results. In NER, each caption error is graded based on severity and the resulting understandability impact. NER vendors decide for themselves what constitutes a critical error, and this subjectivity means that a caption file could receive different NER results depending on who scores the file.

FCC Captioning Compliance

While Canada and the UK use NER as their accuracy metric for compliance, in the USA, the FCC captioning guidelines state, “in order to be accurate, captions must match the spoken words in the dialogue, in their original language (English or Spanish), to the fullest extent possible and include full lyrics when provided on the audio track.” More critically, the guidelines require captions to include all words spoken in the order spoken (i.e., no paraphrasing). Thus, considering the legal requirements of live and recorded captioning, the subjectivity of NER scoring makes it an inferior method for product selection or performance analysis. In fact, the FCC accepted Word Wizards’ CAIV results when IBM went to certify the accuracy of their enterprise live captioning solution for use in U.S. markets on live television.

CAIV Process

CAIV compares the results of speech-to-text products or live human captions to a perfect human transcript produced by one of our expert typists. The process is simple, and the results are 100% objective and based on actual data.

Inputs – Subject content to study, including a caption file and the corresponding media

Outputs – Percentage performance score and a final transcript marked up with all captioning inaccuracies and errors

  1. Receive output from captioning platform
  2. Produce 99.99% accurate human transcript, the “true source”
  3. Word-by-word comparison of “true source” to subject content
  4. Convert results into data-based accuracy score
  5. Analyze results for trends, recurring issues, and performance comparison across multiple products

Learn More About Captioning Analysis Insights and Validation 

Regulatory Governance 

Ensure regulatory compliance including Section 508, ADA, and WCAG

Compliance Monitoring

Monitor the compliance and accuracy of your captions over time

Methodology and Process

Learn about how we use empirical data to understand captioning product performance and text accuracy

Captioning Performance Analysis

Data driven metrics of speech-to-text and captioning performance

Laptop screen displaying a grayscale illustration of a person writing with a quill

Get Started Today!

Word Wizards is ready to be your captioning analysis vendor. Call us today at 301-986-0808 for a free consultation or use our online form to request a quote.