Multi-Speaker 2.0 Benchmarks & Methodology

How does Multi-Speaker 2.0 compare to our previous version and other open-source systems. This page sets out how we measured that — the datasets, the baselines, and the numbers behind every claim, including the places where we are only at parity.

Hear it on Hugging Face MORE ON MULTI-SPEAKER 2.0 API DOCS

Benchmarks run August 2026.

01 — Separation

How clean the tracks are

The separator delivers 32% better separation between voices than 1.0 on broadcast-style speech and 17% better on clean speech — the gain is roughly twice as large on the harder material. With noise in the mix, the complete 2.0 pipeline removes 75% more noise and interference than 1.0's.

Up to 32%

better separation between voices, on Libri2Mix and LRS2-2Mix

75%

more noise and interference removed — Libri2Mix (min) + WHAM!, as SI-SDR improvement

3.9

predicted DNSMOS speech quality, near a point above 1.0

Results on speaker separation

Benchmark	Character of the audio	Less bleed vs 1.0
LRS2-2Mix	Broadcast-style speech, real conditions	32%
Libri2Mix	Clean read speech	17%

Two-speaker benchmarks, 3,000 utterances each, separator in isolation with oracle permutation. Reported as a reduction in bleed rather than an absolute score, because the underlying metric is logarithmic.

Subjective noise removal

Predicted listening quality on the separated tracks reaches 3.9 on a 5-point scale, close to a full point above the previous model, and the background sub-score moves from 3.2 to 4.1 — roughly the difference between background noise you can clearly hear behind the isolated voice and a track that is essentially clean.

This is predicted quality, a model of human ratings, not a human listening panel. It is also referenceless: a track that confidently contains the wrong speaker's words still scores well. We report it alongside the signal-level results, which catch exactly that error, and never on its own.

02 — Diarization

Who spoke when

Measured on its own, Multi-Speaker 2.0's diarizer beats the strongest open-source system we tested, DiariZen. It records lower diarization error than every system we tested, open source and proprietary, in all four domains — and lower than pyannote community-1 on every one of the 14 datasets.

Against the paid proprietary service, we are ahead in all four domains and on 10 of the 14 datasets, but the margins vary: 25% on broadcast, 13% on meetings, 3% on telephone and 0.9% on in-the-wild web video — the last of which we treat as parity rather than a win.

Results by domain

Domain	MS 2.0	pyannote community-1	DiariZen	sortformer-4spk-v2	Paid service
Meetings, far-field + headset (4)	12.00	17.45	15.86	26.76	13.85
Telephone conversations (5)	13.81	22.34	17.18	17.01	14.23
Broadcast / radio (1)	12.63	20.07	14.94	56.87	16.88
In-the-wild web video (4)	23.54	29.15	25.41	36.77	23.76

Diarization error rate %, lower is better, averaged over the corpora in each domain. Open-source comparators: pyannote community-1, DiariZen, sortformer-4spk-v2. Proprietary comparator: a paid diarization service. DIHARD3 is deliberately excluded — pyannote community-1 trains on it and we do not, so including it would not be like-for-like.

Primary-speaker attribution, and non-English telephone

The advantage is not only about detecting simultaneous speech. Recompute DER on primary-speaker annotations — at each instant keep only the speaker who took the floor first, applied identically to the reference and to every system's output, so all overlapped speech is excluded — and the ranking does not change. We stay ahead of pyannote community-1 in all four domains, and ahead of the paid service in three, with a tie on web video.

Non-English telephone conversation is newly covered. The previous diarizer had no conversational telephone speech in its training mix, so this is new domain coverage rather than evidence of better generalization.

CallHome split	Previous	MS 2.0	Fewer errors
Japanese	24.29	14.44	41%
Spanish	21.36	15.14	29%
German	16.15	12.10	25%
Mandarin	14.94	12.50	16%

DER %, lower is better, on the CallHome test splits. Our CallHome results use the free TalkBank release, not the LDC edition most published figures use.

How this was scored

We measure who-spoke-when on 14 public datasets across four domains, scored in the strictest configuration in common use — no forgiveness collar, overlapping speech included, held-out test splits, 16 kHz, applied identically to every system. We compare against three open-source diarizers — pyannote community-1, DiariZen and sortformer-4spk-v2 — and one proprietary paid service.

Those choices matter more than most published DER numbers admit. A 250 ms collar and overlap exclusion, both still common, remove the hardest frames from the denominator.

03 — Downstream transcription

Whether the words survive

The most direct test: transcribe every separated track and count the errors. On LibriCSS, Multi-Speaker 2.0 produces 76% fewer word errors than TF-Locoformer, and 30% fewer than 1.0 on overlapped speech. Up to 20% overlap, accuracy stays within about a point of non-overlapped speech.

76%

fewer word errors than TF-Locoformer

30%

fewer errors than 1.0 on overlapped speech

20%

overlap before accuracy starts to fall

System comparison

Separation front-end	cpWER %
AudioShake Multi-Speaker 2.0	9.17
AudioShake Multi-Speaker 1.0	12.89
MERL TF-Locoformer (Libri2Mix checkpoint)	37.75

Lower is better. LibriCSS, Whisper large-v3, cpWER via the open MeetEval toolkit, identical pipeline for every system. The open-source figure uses a public checkpoint outside its training domain — an accurate picture of what you get off the shelf, not a matched-training claim.

How accuracy holds as overlap increases

“20% overlap” means a fifth of total speech time has two or more people talking at once. It is a property of the recording, not of any single interruption.

LibriCSS condition	cpWER %
0S — no overlap, short silences	5.81
10% overlap	7.04
20% overlap	7.18
30% overlap	13.66
40% overlap	10.03

Conditions differ in more than overlap ratio — the two 0% conditions differ in silence length — so this is not a monotonic curve. See failure modes for the 30% and 40% conditions. No denoise frontend was enabled in these runs, so they measure the separator and diarizer rather than the complete shipping path.

Compared across our models

The overlap conditions above, run across all three systems rather than ours alone. Scored with ORC-WER, which counts transcription errors without requiring that each word be assigned to the right speaker — a different measurement from the cpWER figures elsewhere on this page, and not directly comparable to them.

ORC-WER %	10% overlap	20%	30%	40%
No separation	10.1	14.0	19.6	27.8
Multi-Speaker 1.0	10.2	15.1	17.7	18.8
AudioShake Multi-Speaker 2.0	9.4	9.8	12.8	14.1

Lower is better. Every arm uses the same recordings and the same transcription model — the only thing that changes is what runs in front of it.

Measured on LibriCSS, a standard benchmark of recorded multi-speaker sessions with controlled amounts of overlapping speech.
Speaker attribution is measured separately — see the diarization section.
LibriCSS has no condition above 40% overlap, so the table stops there. Nothing here should be read as a projection beyond that point.
Zero-overlap conditions are excluded. This table is about overlapped speech, and separation is not the right tool for recordings that have none.

04 — Structured output

What a job returns

One audio track per speaker at the input sample rate, an overlap-aware timeline of who spoke when, and two confidence scores — each on a 0-to-1 scale, returned per 20 ms frame and as a whole-file value weighted by speech activity, so silence cannot inflate them. Input can be anything from 8 kHz to 48 kHz, mono or dual mono; true stereo follows later this year.

assignment_confidence

Did this speech go to the right speaker?

The label-quality signal. Low values mean the audio may be clean but the identity behind it is unreliable — fine for cleanup, risky for anything routed by speaker.

separation_confidence

Did the overlapping voices come apart?

Most informative where people actually talk over each other. Low values mean a track probably carries bleed from another voice — fine for talk-time analytics, not for voice cloning or per-speaker ASR.

The two scores catch different failures and one number cannot tell them apart — audio can be cleanly separated but attributed to the wrong person, or correctly attributed and still carry bleed. If you need a single value, take the lower of the two.

Reading the two scores together

Separation	Assignment	What it means
low	low	Both collapsed, usually on overlapping speech. Neither the audio nor the labels are trustworthy — drop, or route to review.
high	high	The strongest “trust this” signal. Also what you see where there is no speech at all.
high	low	Little to separate, but speaker identity is unreliable. Fine for cleanup; risky for anything routed by speaker.
low	high	Labels are trustworthy but the audio likely carries bleed. Fine for talk-time analytics; not for voice cloning or per-speaker ASR.

We deliberately do not publish recommended thresholds. Score distributions vary across content types and the right cutoff depends on the application, so treat them as a relative ranking signal rather than a gate. For guidance on using them in code, see the API documentation.

05 — Failure modes

Where it stops working

Sustained overlap above 20%

Separation quality falls off past about a fifth of speech time in crosstalk — the 30% and 40% LibriCSS conditions show it.

Cast size

There is no fixed ceiling on speaker count, but accuracy falls as the cast grows — speaker-count error is more than five times higher at five or six speakers than at two. Pass a known count where you have one.

Correct labels, contaminated audio

Attribution errors and residual bleed are independent failures; the two confidence scores tell them apart.

Metric caveats

Predicted listening quality is referenceless, and several diarization corpora are in-domain for us and for the systems we compare against.

06 — DEFINITIONS

What each number measures

Every metric on this page, defined in the form we score it. We report two families because neither is sufficient on its own: signal-level metrics ask whether the recovered audio is clean, task-level metrics ask whether it still works for the systems that consume it. SI-SDR rewards waveform fidelity a recognizer may not care about, and ASR-based metrics tolerate artifacts a person would not accept.

DER

Diarization error rate

The share of speech time attributed to the wrong speaker, missed entirely, or invented. Lower is better. We score with no forgiveness collar and overlapping speech included.

cpWER

Concatenated minimum-permutation word error rate

Transcribe every output track, then score under the speaker assignment that minimizes total error. It penalizes both transcription mistakes and speaker-attribution mistakes, which is why it is the closest proxy for what a real pipeline experiences.

Speaker-count error

How often the number of people is wrong

The absolute difference between speakers found and speakers present, divided by speakers present, averaged per recording. A four-speaker file where a system finds three scores 25%.

Overlap ratio

What “20% overlap” means

The share of total speech time in which two or more people are talking simultaneously. At 20%, a fifth of the speech in the recording is crosstalk. It is a property of the recording, not of any single interruption.

ORC-WER

Optimal reference combination word error rate

Counts transcription errors without requiring that each word be assigned to the right speaker. It isolates whether the words survived separation, where cpWER also penalizes getting the speaker wrong. The two are not directly comparable, which is why every table on this page names the one it uses.

SI-SDR

Scale-invariant signal-to-distortion ratio

Measures how much of a separated track is the target voice against everything else in it — the other speaker, noise, artifacts. The scale is logarithmic, so we report changes as a percentage reduction in that unwanted signal, which we call bleed, rather than as an absolute score.

DNSMOS

Predicted listening quality, 1 to 5

A model trained to predict how human listeners would rate a recording. It needs no clean reference, which also means it will score a track that confidently contains the wrong speaker's words highly. We never report it on its own.