AudioShake vs.
Demucs, BS-Roformer & MVSep
Music separation vs. Demucs & BS-Roformer
AudioShake leads the field in high-quality source separation, which is typically measured via something called the Signal-to-Distortion (SDR) score. AudioShake has repeatedly demonstrated its ability to achieve the highest SDR scores and set several state of the art benchmarks.
But SDR scores alone are not a good way to measure source separation–it's quite possible to achieve a high SDR score while producing output that doesn't sound very good. That's why we focus even more on perceptual quality. In Meta's SAM Audio evaluation, AudioShake was the highest-performing discriminative model on perceptual quality, ahead of Moises / Music.ai, FADR, Lalal and MossFormer. AudioLabs Erlangen reached the same conclusion on piano concertos, finding AudioShake produced the highest-quality piano and orchestral stems.
The music information retrieval community typically uses something called the Signal-to-Distortion (SDR) score to measure quality, and AudioShake has repeatedly demonstrated its ability to achieve the highest SDR scores. However, Even though we hold many of the state-of-the-art benchmarks, we caution people from solely using SDR score as the best indicator of a high-quality model, because it is quite possible to achieve a high SDR score while the actual results sound poor.
That's why we have developed perceptual metrics that we use for evaluating model performance, and sometimes choose lower-SDR scores models that perform better on different tasks.
With all that said, below is a sample of our stem separation scores for music.
Hear the difference: vocal isolation compared
Speech separation: Multi-Speaker 2.0 vs. open-source baselines
The comparisons above cover music. Speech is measured separately: Multi-Speaker 2.0 against Multi-Speaker 1.0 and open-source baselines on separation, diarization, and downstream transcription, with the datasets and methodology written out.
Lyric transcription accuracy (WER)
To evaluate transcription models, researchers use industry-wide benchmarks that measure word error rate (WER) alongside punctuation and formatting metrics. Alignment is evaluated separately, measuring how closely predicted word and line timings match the audio. At ISMIR, AudioShake’s research team presented a new benchmark based on the JamendoLyrics dataset that captures the finer nuances of written lyrics, called Jam-ALT.
Below are the results of running various transcription and alignment systems against these benchmarks. AudioShake sets the state-of-the-art in both categories — including 95% accuracy on lyric alignment — making it the most accurate system available for transcription and for the word-level timing that synced lyrics, karaoke, and captioning depend on.
