Zero WER Can Hide Speaker Errors: A cpWER Acceptance Test

A transcript can contain every correct word and still put those words under the wrong speaker. A word error rate computed after discarding speaker labels cannot catch that mistake. Concatenated minimum-permutation word error rate, or cpWER, compares speaker-grouped word sequences after choosing one best global mapping between reference speakers and hypothesis labels.

That mapping matters. Renaming every instance of speaker A to B and every B to A is harmless for anonymous diarization labels. Changing the mapping halfway through a recording is not. Our eight-case synthetic test produced pooled WER of zero for both, but cpWER of zero for the global rename and 33.3% for the inconsistent labels.

Use the fixture below to check a scoring pipeline before using a meeting transcript to assign work. It tests the scorer, not a speech-recognition provider. There is no audio, real participant identification, provider accuracy ranking or production action in this experiment.

Start with the words and the grouping

Our reference contains four non-overlapping synthetic segments:

There are twelve reference words. The pooled lexical calculation sorts the segments by start time and joins their text into one string. That operation deliberately throws away the speaker labels. A perfect pooled score therefore says nothing about whether the invoice instruction belongs to A or B.

The cpWER calculation preserves speaker grouping. MeetEval's algorithm documentation describes computing distances between speaker streams and choosing the minimum-cost global assignment. Streams from missing or extra speakers are matched against empty streams. The denominator remains the number of reference words; this is a word-based score, not a percentage of incorrectly labeled seconds.

After defining the two scores, run the same safe inputs in Code Interpreter Tools. Keep the output's word counts and selected assignment, not just the headline percentage.

What the controlled cases showed

We ran Python 3.13.14 and MeetEval 0.4.3 through Aident Loadout's current Code Interpreter Action on October 2, 2026. All eight cases passed their three assertions, plus the pinned-version and unchanged-reference checks: 26 assertions in total, with no error or stderr.

The four errors in the late-flip case are not four wrongly attributed words counted directly. They are the edit operations selected after speaker-stream concatenation and global assignment. Do not subtract pooled WER from cpWER and call the difference a calibrated diarization error rate.

For the global rename, the returned assignment maps reference A to hypothesis B and reference B to hypothesis A. This is not evidence that either label identifies a real person. Anonymous cluster labels and participant identities are separate contracts.

The missing-speaker case reports one missed speaker; the extra-stream case reports one false-alarm speaker. Those observations belong to this fixture. They are not estimates of a provider's missed-speaker rate.

Run the complete fixture

Use a disposable environment with meeteval==0.4.3 installed. The current MeetEval Python interface exposes the two low-level scoring functions used here. If your environment uses a different version, record it and review changed behavior rather than deleting the assertion.

For managed execution, tell your coding agent:

Follow https://aident.ai/SETUP.md

Select Code Interpreter Tools, inspect the current Action schema and preflight estimate, and provide the complete fixture. If its disposable runtime does not yet contain the dependency, install the pinned package there with the runtime's supported package mechanism. Do not install it into a production application merely to run this test.

The code operates on in-memory synthetic segments. It does not read recordings, local files, customer data or credentials, and it makes no transcription request.

import copy
import importlib.metadata
import json
import sys

import meeteval
from meeteval.wer.wer.cp import cp_word_error_rate
from meeteval.wer.wer.siso import siso_word_error_rate

assert importlib.metadata.version("meeteval") == "0.4.3"
reference = [
    ("A", 0, 1, "approve the budget"),
    ("B", 1, 2, "send the invoice"),
    ("A", 2, 3, "schedule the review"),
    ("B", 3, 4, "notify the supplier"),
]
original = copy.deepcopy(reference)


def stm(segments):
    return meeteval.io.STM.parse("\n".join(
        f"fixture 1 {speaker} {start} {end} {words}"
        for speaker, start, end, words in segments
    ))


def pooled(segments):
    return " ".join(s[3] for s in sorted(segments, key=lambda s: s[1]))


rename = lambda s: "B" if s == "A" else "A"
late_flip = [(rename(s) if t >= 2 else s, t, e, w)
             for s, t, e, w in reference]
repeated = [(s, t, e, "yes") for s, t, e, _ in reference]
repeated_flip = [(rename(s) if t >= 2 else s, t, e, w)
                 for s, t, e, w in repeated]
cases = [
    ("identical", reference, reference, 0, 0),
    ("global_rename", reference,
     [(rename(s), t, e, w) for s, t, e, w in reference], 0, 0),
    ("late_label_flip", reference, late_flip, 0, 4),
    ("all_one_speaker", reference,
     [("X", t, e, w) for s, t, e, w in reference], 0, 12),
    ("extra_speaker", reference,
     [("C" if t == 3 else s, t, e, w) for s, t, e, w in reference], 0, 6),
    ("one_word_substitution", reference,
     [(s, t, e, w.replace("budget", "project"))
      for s, t, e, w in reference], 1, 1),
    ("timestamps_shifted_100s", reference,
     [(s, t + 100, e + 100, w) for s, t, e, w in reference], 0, 0),
    ("repeated_words_late_flip", repeated, repeated_flip, 0, 0),
]
rows = []
for name, ref, hyp, expected_pooled, expected_cp in cases:
    lexical = siso_word_error_rate(pooled(ref), pooled(hyp))
    cp = cp_word_error_rate(stm(ref), stm(hyp))
    length = sum(len(s[3].split()) for s in ref)
    assert lexical.errors == expected_pooled, (name, lexical)
    assert cp.errors == expected_cp, (name, cp)
    assert lexical.length == cp.length == length, (name, length)
    rows.append({
        "case": name, "reference_words": length,
        "pooled_errors": lexical.errors, "pooled_wer": lexical.error_rate,
        "cp_errors": cp.errors, "cpwer": cp.error_rate,
        "assignment": cp.assignment,
        "missed_speaker": cp.missed_speaker,
        "false_alarm_speaker": cp.falarm_speaker,
    })
assert reference == original
print(json.dumps({
    "python": sys.version, "meeteval": importlib.metadata.version("meeteval"),
    "cases": rows, "assertions": 26, "status": "PASS",
}, indent=2))

Expected terminal output contains status: PASS, meeteval: 0.4.3, eight case rows and assertions: 26. The late-label-flip row has cp_errors: 4 and cpwer: 0.3333333333333333. The pooled score remains zero.

Inspect each returned assignment. Ties can admit more than one equally good assignment; do not make a downstream identity decision from which optimal tie the library returns.

Two zero-score counterexamples to keep

The shifted-timestamp row is intentionally wrong by 100 seconds. cpWER can still be zero because this calculation preserves sequence order without imposing reference/hypothesis temporal overlap. The MeetEval paper introduces time-constrained matching for this separate concern. If timing matters, define that metric and its collar explicitly, then test it. This article does not execute tcpWER or claim a tested collar.

The repeated-word row is a stronger warning against treating cpWER as complete speaker validation. Both reference speakers say “yes” twice. Swapping the last two labels leaves each concatenated word sequence unchanged, so cpWER cannot distinguish the mistake in this fixture.

For a real diarization evaluation, retain time-aligned ground truth, scoring regions, boundary-collar and overlap policy, and a named time-based scoring implementation. MeetEval also exposes DER interfaces, but no DER value was measured here. Timing and speaker identity cannot be inferred from the word-based results.

These examples are acceptance tests for specific failure modes, not representative meeting data. They establish no population-level accuracy, language coverage, noise robustness, latency or cost advantage.

Put the scorer before the consumer

This choice comes up in practice. A May 2026 upstream discussion asks how to separate recognition errors from diarization when scoring meeting transcripts. It is useful problem context, not a current product defect or a benchmark result.

Before comparing provider outputs, write down what the consumer needs:

• Correct words, with a fixed tokenization and text-normalization policy.

• Consistent anonymous speaker grouping across the entire recording.

• Timing precise enough for the chosen downstream job.

• Participant-reviewed mapping from anonymous labels to real identities.

Keep the reference, raw hypothesis, normalization policy, scorer version, errors, denominator and selected assignment together. For a dataset, aggregate errors and reference lengths; do not average percentages from short and long recordings equally.

If the next step is remote execution, code interpreter versus AI-agent sandbox explains the boundary. A passing remote fixture does not imply access to your local meeting application.

When choosing a transcription Action, the Gladia versus Deepgram contract comparison addresses exposed controls, outputs and approval boundaries. This scoring test is complementary; it does not replace equal audio fixtures or establish a winning provider.

Run the speaker-scoring acceptance fixture before trusting a single transcript-quality percentage. Then evaluate real, consented recordings separately, and require a human to confirm who should own any action item.

Sources and refresh trigger

The linked MeetEval algorithm documentation supplies the global-assignment definition; its Python interface supplies the API contract. The linked toolkit paper explains time-constrained matching, and upstream issue 129 supplies dated practitioner context. The numerical results come from our October 2, 2026 synthetic fixture, not those sources or a provider leaderboard.

Re-run and review this fixture when adopting another MeetEval version, changing tokenization or normalization, adding time constraints, or changing the transcript-to-speaker grouping adapter. Keep the old version and results with the new comparison; a zero score is never a substitute for participant-confirmed identity.