Balanced Datasets Are Not Enough: Estimating and Mitigating Gender Bias in Deep Image Representations
News Release Summary
Researchers from the University of Virginia, UCLA, and the Allen Institute for Artificial Intelligence have found that simply balancing training datasets is not enough to prevent AI image recognition systems from amplifying gender stereotypes. The team studied models trained on two widely used datasets — COCO, which labels everyday objects, and imSitu, which labels human actions — and found that even when they artificially rebalanced the data so that men and women appeared equally often alongside each category label, the trained models still learned to associate gender with those categories at roughly the same inflated rates as models trained on the original skewed data. To measure this problem more precisely, the researchers developed two new metrics they call "dataset leakage" and "model leakage," which quantify how accurately an outside observer could guess the gender of a person in an image simply by looking at what labels a model assigns to it; the gap between those two measures captures how much extra gender information the model is smuggling into its predictions beyond what the data itself contains. Their explanation for why balancing fails is straightforward: datasets contain countless unlabeled visual cues — like the presence of children, clothing styles, or body posture — that correlate with gender and cannot be neutralized by adjusting label counts alone. To actually reduce the bias, the team trained models with an adversarial component that actively penalizes the network for encoding gender-predictable features in its intermediate layers, achieving a 53 to 67 percent reduction in bias amplification while sacrificing only about one to two percentage points in classification accuracy. The work is a caution to anyone who assumes that demographic fairness in AI can be achieved through dataset curation alone, and it points toward deeper architectural interventions as a more reliable path forward.
abstract
In this work, we present a framework to measure and mitigate intrinsic biases with respect to protected variables --such as gender-- in visual recognition tasks. We show that trained models significantly amplify the association of target labels with gender beyond what one would expect from biased datasets. Surprisingly, we show that even when datasets are balanced such that each label co-occurs equally with each gender, learned models amplify the association between labels and gender, as much as if data had not been balanced! To mitigate this, we adopt an adversarial approach to remove unwanted features corresponding to protected variables from intermediate representations in a deep neural network -- and provide a detailed analysis of its effectiveness. Experiments on two datasets: the COCO dataset (objects), and the imSitu dataset (actions), show reductions in gender bias amplification while maintaining most of the accuracy of the original models.
details
citation
@inproceedings{wang2019balanced,
title = {Balanced Datasets Are Not Enough: Estimating and Mitigating Gender Bias in Deep Image Representations},
author = {Wang, Tianlu and Zhao, Jieyu and Yatskar, Mark and Chang, Kai-Wei and Ordonez, Vicente},
year = {2019},
booktitle = {International Conference on Computer Vision. ICCV 2019},
url = {https://arxiv.org/abs/1811.08489},
}