Multi-Speaker 2.0 Benchmarks & Methodology
Multi-Speaker 2.0 separates speakers even in moments of overlap.
Benchmarks run August 2026. Comparator versions are named in each section.
01 — Diarization
Who spoke when
The strongest open-source diarizer we tested is diarizen. Multi-Speaker 2.0 still records lower diarization error than every system we tested, open source and proprietary, in all four domains — and lower than community-1 on every one of the 14 datasets.
4 / 4
domains ahead of every system tested
7–24%
lower DER than the best open source, by domain
14 / 14
datasets ahead of community-1
Corpora by domain — Meetings: AMI, AMI-SDM, AISHELL4-SDM, AliMeeting-SDM. Telephone: CallHome EN/DE/ES/ZH/JA. Broadcast: TAL. Web video: VoxConverse, VoxMM, MSDWild, AVA-AVD.
Against precision-2, the proprietary comparator, we are ahead in all four domains and on 10 of the 14 datasets, but the margins vary: 25% on broadcast, 13% on meetings, 3% on telephone and 0.9% on in-the-wild web video — the last of which we treat as parity rather than a win.
Results by domain
| Domain | MS 2.0 | community-1 | diarizen | sortformer-4spk-v2 | precision-2 |
|---|---|---|---|---|---|
| Meetings, far-field + headset (4) | 12.00 | 17.45 | 15.86 | 26.76 | 13.85 |
| Telephone conversations (5) | 13.81 | 22.34 | 17.18 | 17.01 | 14.23 |
| Broadcast / radio (1) | 12.63 | 20.07 | 14.94 | 56.87 | 16.88 |
| In-the-wild web video (4) | 23.54 | 29.15 | 25.41 | 36.77 | 23.76 |
Diarization error rate %, lower is better, averaged over the corpora in each domain. Open-source comparators: community-1, diarizen, sortformer-4spk-v2. Proprietary comparator: precision-2. DIHARD3 is deliberately excluded — community-1 trains on it and we do not, so including it would not be like-for-like.
Primary-speaker attribution, and non-English telephone
Recompute error keeping only the speaker who took the floor first at each instant, so all overlapped speech is excluded. The ranking holds: we stay ahead of community-1 in all four domains, and ahead of precision-2 in three, with an exact tie on web video.
| CallHome split | Previous | MS 2.0 | Fewer errors |
|---|---|---|---|
| Japanese | 24.29 | 14.44 | 41% |
| Spanish | 21.36 | 15.14 | 29% |
| German | 16.15 | 12.10 | 25% |
| Mandarin | 14.94 | 12.50 | 16% |
The previous diarizer had no conversational telephone speech in its training mix, so this is new domain coverage rather than better generalization. Our CallHome results use the free TalkBank release, not the LDC edition most published figures use.
How this was scored
No forgiveness collar. Overlapping speech included. Held-out test splits only, at 16 kHz — the model itself runs natively from 8 kHz to 48 kHz, but 16 kHz is what these public corpora provide and what every system is scored at here. Diarizer evaluated in isolation — no denoise frontend, no separator in the path. Comparators: community-1, diarizen, sortformer-4spk-v2 and precision-2, each scored identically.
A 250 ms collar and the exclusion of overlapping speech are both still common in published figures. Both remove the hardest frames from the denominator, so numbers scored that way are not comparable to these.
02 — Speaker counting
How many people are talking
Across 1,172 recordings we count speakers about 16% more accurately than community-1. The useful part is the shape: below three speakers the two systems are level, and from three speakers up our margin widens to about 18%.
16%
more accurate than community-1, all recordings
18%
better than community-1 on 3+ speakers
1,172
recordings scored
Results by speaker count
| Speakers present | Recordings | MS 2.0 | community-1 | diarizen (pending) | sortformer-4spk-v2 (pending) | precision-2 (pending) |
|---|---|---|---|---|---|---|
| 1–2 | 387 | 6.07 | 5.68 | — | — | — |
| 3–4 | 335 | 21.47 | 27.04 | — | — | — |
| 5–6 | 205 | 33.38 | 40.47 | — | — | — |
| 7 or more | 245 | 24.77 | 29.90 | — | — | — |
| All recordings | 1,172 | 19.16 | 22.93 | — | — | — |
Speaker-count error %, weighted per recording. A four-speaker file where a system finds three scores 25%. Only community-1 has been scored on this metric so far; the diarizen, sortformer-4spk-v2 and precision-2 columns are awaiting runs. The 1–2 row is published rather than hidden: community-1 edges us there by four tenths of a point, and the margin turns in our favour from three speakers up.
03 — Separation
How clean the tracks are
The separator leaves 32% less bleed between voices than 1.0 on broadcast-style speech and 17% less on clean read speech — the gain is roughly twice as large on the harder material. With noise in the mix, the full 2.0 path removes 75% more noise and interference than 1.0's.
32%
less bleed between voices, broadcast-style speech
75%
more noise and interference removed
3.9
predicted listening quality, near a point above 1.0
Results by benchmark
| Benchmark | Character of the audio | Less bleed vs 1.0 | Vs open source (pending) |
|---|---|---|---|
| LRS2-2Mix | Broadcast-style speech, real conditions | 32% | — |
| Libri2Mix | Clean read speech | 17% | — |
Two-speaker benchmarks, 3,000 utterances each, separator in isolation with oracle permutation. Reported as a reduction in bleed rather than an absolute score, because the underlying metric is logarithmic. The open-source comparison column is awaiting MossFormer2 and SepFormer runs.
A note on perceptual quality
Predicted listening quality on the separated tracks reaches 3.9 on a 5-point scale, close to a full point above the previous model, and the background sub-score moves from 3.2 to 4.1 — roughly the difference between background noise you can clearly hear behind the isolated voice and a track that is essentially clean.
This is predicted quality, a model of human ratings, not a human listening panel. It is also referenceless: a track that confidently contains the wrong speaker's words still scores well. We report it alongside the signal-level results, which catch exactly that error, and never on its own.
04 — Downstream transcription
Whether the words survive
The most direct test: transcribe every separated track and count the errors. Multi-Speaker 2.0 produces 4.1× fewer word errors than TF-Locoformer, and 30% fewer than 1.0 on overlapped speech. Up to 20% overlap, accuracy stays within about a point of non-overlapped speech.
4.1×
fewer word errors than TF-Locoformer
30%
fewer errors than 1.0 on overlapped speech
20%
overlap before accuracy starts to fall
System comparison
| Separation front-end | cpWER % |
|---|---|
| AudioShake Multi-Speaker 2.0 | 9.17 |
| Multi-Speaker 1.0 separator | 12.89 |
| MERL TF-Locoformer (Libri2Mix checkpoint) | 37.75 |
| MossFormer2 (open source, pending) | — |
| SepFormer (open source, pending) | — |
Lower is better. LibriCSS, Whisper large-v3, cpWER via the open MeetEval toolkit, identical pipeline for every system. The open-source figure uses a public checkpoint outside its training domain — an accurate picture of what you get off the shelf, not a matched-training claim.
How accuracy holds as overlap increases
“20% overlap” means a fifth of total speech time has two or more people talking at once. It is a property of the recording, not of any single interruption.
| LibriCSS condition | cpWER % |
|---|---|
| 0S — no overlap, short silences | 5.81 |
| 0L — no overlap, long silences | 10.89 |
| 10% overlap | 7.04 |
| 20% overlap | 7.18 |
| 30% overlap | 13.66 |
| 40% overlap | 10.03 |
Conditions differ in more than overlap ratio — the two 0% conditions differ in silence length — so this is not a monotonic curve. See failure modes for the 30% and 40% conditions. No denoise frontend was enabled in these runs, so they measure the separator and diarizer rather than the complete shipping path.
Compared across our models
The overlap conditions above, run across all three systems rather than ours alone. Scored with ORC-WER, which counts transcription errors without requiring that each word be assigned to the right speaker — a different measurement from the cpWER figures elsewhere on this page, and not directly comparable to them.
| ORC-WER % | 10% overlap | 20% | 30% | 40% |
|---|---|---|---|---|
| No separation | 10.1 | 14.0 | 19.6 | 27.8 |
| Multi-Speaker 1.0 | 10.2 | 15.1 | 17.7 | 18.8 |
| AudioShake Multi-Speaker 2.0 | 9.4 | 9.4 | 12.8 | 14.1 |
Lower is better. Every arm uses the same recordings and the same transcription model — the only thing that changes is what runs in front of it.
- Measured on LibriCSS, a standard benchmark of recorded multi-speaker sessions with controlled amounts of overlapping speech.
- Speaker attribution is measured separately — see the diarization section.
- LibriCSS has no condition above 40% overlap, so the table stops there. Nothing here should be read as a projection beyond that point.
- Zero-overlap conditions are excluded. This table is about overlapped speech, and separation is not the right tool for recordings that have none.
- Research evaluation pipeline, not the production stack. Figures are subject to change.
05 — Structured output
What a job returns
One audio track per speaker at the input sample rate, an overlap-aware timeline of who spoke when, and two confidence scores — returned per 20 ms frame and as a whole-file value weighted by speech activity, so silence cannot inflate them.
assignment_confidence
Did this speech go to the right speaker?
The label-quality signal. Low values mean the audio may be clean but the identity behind it is unreliable — fine for cleanup, risky for anything routed by speaker.
separation_confidence
Did the overlapping voices come apart?
Most informative where people actually talk over each other. Low values mean a track probably carries bleed from another voice — fine for talk-time analytics, not for voice cloning or per-speaker ASR.
The two scores catch different failures and one number cannot tell them apart — audio can be cleanly separated but attributed to the wrong person, or correctly attributed and still carry bleed. If you need a single value, take the lower of the two.
Reading the two scores together
| Separation | Assignment | What it means |
|---|---|---|
| low | low | Both collapsed, usually on overlapping speech. Neither the audio nor the labels are trustworthy — drop, or route to review. |
| high | high | The strongest “trust this” signal. Also what you see where there is no speech at all. |
| high | low | Little to separate, but speaker identity is unreliable. Fine for cleanup; risky for anything routed by speaker. |
| low | high | Labels are trustworthy but the audio likely carries bleed. Fine for talk-time analytics; not for voice cloning or per-speaker ASR. |
We deliberately do not publish recommended thresholds. Score distributions vary across content types and the right cutoff depends on the application, so treat them as a relative ranking signal rather than a gate. For guidance on using them in code, see the API documentation.
06 — Failure modes
Where it stops working
01
Sustained overlap above 20%
Separation quality falls off past about a fifth of speech time in crosstalk — the 30% and 40% LibriCSS conditions show it.
02
Cast size
Speaker-count error roughly triples from two speakers to five or six; pass a known count where you have one.
03
Correct labels, contaminated audio
Attribution errors and residual bleed are independent failures; the two confidence scores tell them apart.
04
Metric caveats
Predicted listening quality is referenceless, and several diarization corpora are in-domain for us and for the systems we compare against.
07 — DEFINITIONS
What each number measures
Every metric on this page, defined in the form we score it. We report two families because neither is sufficient on its own: signal-level metrics ask whether the recovered audio is clean, task-level metrics ask whether it still works for the systems that consume it. SI-SDR rewards waveform fidelity a recognizer may not care about, and ASR-based metrics tolerate artifacts a person would not accept.
DER
Diarization error rate
The share of speech time attributed to the wrong speaker, missed entirely, or invented. Lower is better. We score with no forgiveness collar and overlapping speech included.
cpWER
Concatenated minimum-permutation word error rate
Transcribe every output track, then score under the speaker assignment that minimizes total error. It penalizes both transcription mistakes and speaker-attribution mistakes, which is why it is the closest proxy for what a real pipeline experiences.
Speaker-count error
How often the number of people is wrong
The absolute difference between speakers found and speakers present, divided by speakers present, averaged per recording. A four-speaker file where a system finds three scores 25%.
Overlap ratio
What “20% overlap” means
The share of total speech time in which two or more people are talking simultaneously. At 20%, a fifth of the speech in the recording is crosstalk. It is a property of the recording, not of any single interruption.
ORC-WER
Optimal reference combination word error rate
Counts transcription errors without requiring that each word be assigned to the right speaker. It isolates whether the words survived separation, where cpWER also penalizes getting the speaker wrong. The two are not directly comparable, which is why every table on this page names the one it uses.
SI-SDR
Scale-invariant signal-to-distortion ratio
Measures how much of a separated track is the target voice against everything else in it — the other speaker, noise, artifacts. The scale is logarithmic, so we report changes as a percentage reduction in that unwanted signal, which we call bleed, rather than as an absolute score.
DNSMOS
Predicted listening quality, 1 to 5
A model trained to predict how human listeners would rate a recording. It needs no clean reference, which also means it will score a track that confidently contains the wrong speaker's words highly. We never report it on its own.
Get in touch.