Logo image
Correlation-based methods for representative fairness metric selection: An empirical study on efficiency and caveats in model evaluation
Journal article   Open access   Peer reviewed

Correlation-based methods for representative fairness metric selection: An empirical study on efficiency and caveats in model evaluation

Rafael B. Loureiro, Tiago P. Pagano, Fernanda V.N. Lisboa, Lian F.S. Nascimento, Ewerton L.S. Oliveira, Ingrid Winkler and Erick G. Sperandio Nascimento
Expert Systems with Applications, Vol.268, 126344
05/04/2025

Abstract

Bias Correlation Representative metric Unfairness
Addressing bias and unfairness in machine learning models in different application domains is a multifaceted challenge. Despite the variety of fairness metrics available, identifying an optimal set for evaluating a model’s fairness is still an open question due to the diverse nature of these metrics and the lack of a comprehensive approach for ensuring fairness across different applications. This study aims to propose a method that allows the selection of the most representative metrics for bias and fairness assessment, in post-processing for machine learning models in different contexts. We delve into the use of a correlation-based strategy as a heuristic for fairness metric selection, applying bootstrap sampling using the Markov chain Monte Carlo technique, with our proposed improvements, including stratified sampling, stopping criterion, and Kendall correlation, to address the data bias representation, the computational cost, and the robustness, respectively. An average decrease of 64.37% in the number of models and of 20.00% in processing time was achieved. Moreover, the proposed method effectively paired metrics with similar behaviour, highlighting the presence of a similar term as a strong indicator of a direct relationship. While no standout metric emerges across all contexts, within specific models or datasets, certain metrics consistently stand out. In a complex scenario using a large language model for sexism detection, the proposed method achieved a 71.93% reduction in execution time while forming more comprehensive metric groups. Overall, the proposed method successfully selects the representative metric with a considerable gain in computational costs, demonstrating its practicality for real-world applications. [Display omitted] •Validate correlation for finding a representative fairness metric in a context.•Reduce the computational power needed to find the representative metrics.•Validate the method empirically, showing effectiveness across models and datasets.•Analyse fairness metrics context relations and dependencies.
url
https://doi.org/10.1016/j.eswa.2024.126344View
Published (Version of record) Open CC BY V4.0

Metrics

Details

Logo image

Usage Policy