To evaluate the accuracy of transcription models, researchers use industry-wide benchmarks to evaluate the word error rate (WER), as well as punctuation and formatting related metrics.  At ISMIR, AudioShake's Research team presented a new benchmark based on the JamendoLyrics dataset that accounted for the finer nuances of written lyrics called Jam-ALT.

Below are the results of running various transcription systems against this new benchmark for evaluation. AudioShake has set the State of the Art benchmarks across both lyric transcription and alignment.

The music information retrieval community typically uses something called the Signal-to-Distortion (SDR) score to measure quality, and AudioShake has repeatedly demonstrated its ability to achieve the highest SDR scores. However, Even though we hold many of the state-of-the-art benchmarks, we caution people from solely using SDR score as the best indicator of a high-quality model, because it is quite possible to achieve a high SDR score while the actual results sound poor.

That's why we have developed perceptual metrics that we use for evaluating model performance, and sometimes choose lower-SDR scores models that perform better on different tasks.

With all that said, below is a sample of our stem separation scores for music.