Abstract
Traditional Blind Source Separation Evaluation (BSS-Eval)
metrics were originally designed to evaluate linear audio source separation
models based on methods such as time-frequency masking. However,
recent generative models may introduce nonlinear relationships between
the separated and reference signals, limiting the reliability of these
metrics for objective evaluation. To address this issue, we conduct a
Degradation Category Rating listening test and analyze correlations
between the obtained degradation mean opinion scores (DMOS) and a set
of objective audio quality metrics for the task of singing voice separation.
We evaluate three state-of-the-art discriminative models and two new,
competitive generative models. For both discriminative and generative
models, intrusive embedding-based metrics show higher correlations with
DMOS than conventional intrusive metrics such as BSS-Eval metrics.
For discriminative models, the highest correlation is achieved by the
MSE computed on Music2Latent embeddings. When it comes to the
evaluation of generative models, the strongest correlations are evident for
the multi-resolution STFT loss and the MSE calculated on MERT-L12
embeddings, with the latter also providing the most balanced correlation
across both model types. Our results highlight the limitations of BSS-Eval
metrics for evaluating generative singing voice separation models and
emphasize the need for careful selection and validation of alternative
evaluation metrics for the task of singing voice separation.