Discovering Trade-offs in Fairness and Accuracy: A Multi-Objective Approach

Abstract

In recent years, there has been a shift from focusing exclusively on the accuracy of machine learning systems to a more holistic and human-centered approach that includes privacy, fairness, transparency and more. Many of these dimensions are often considered to conflict with each other. For example, there can be a trade-off between the accuracy and fairness of a predictive model. In fairness analysis, the aim is to establish that machine learning models do not discriminate based on protected or sensitive characteristics such as race, gender, age, or religion. In practice there are many alternative notions of fairness, some of which themselves may not be mutually compatible. In this paper, we explore this relationship between accuracy and different notions of fairness using German Credit dataset, where training a model using standard techniques has been shown to lead to biased predictions. We explore the trade-off between accuracy and six different fairness metrics using a multi-objective training approach, which aims to maximize both accuracy and fairness. Our results show that in certain cases, there exists a trade-off between accuracy and different notions of fairness. In these cases, the multi-objective approach provides a set of models that balance the trade-off in different ways. Further, in other cases, the approach does not lead to a trade-off, instead giving rise to a model that is both accurate and fair simultaneously, when this was not achieved using a single-objective approach. Therefore, we show that by explicitly targeting fairness during training, decision makers can have access to a range of models that might meet their accuracy and fairness requirements. Moreover, we also show that a multi-objective approach identifies situations where an assumed trade-off between fairness and accuracy need not exist.