Loading...

Decompose-and-Compose: A Compositional Approach to Mitigating Spurious Correlation

Noohdani, F.H ; Sharif University of Technology | 2024

43 Viewed
  1. Type of Document: Article
  2. DOI: 10.1109/CVPR52733.2024.02612
  3. Publisher: 2024
  4. Abstract:
  5. While standard Empirical Risk Minimization (ERM) training is proven effective for image classification on in-distribution data, it fails to perform well on out-of-distribution samples. One of the main sources of distribution shift for image classification is the compositional nature of images. Specifically, in addition to the main object or component(s) determining the label, some other image components usually exist, which may lead to the shift of input distribution between train and test environments. More importantly, these components may have spurious corre-lations with the label. To address this issue, we propose Decompose-and-Compose (DaC), which improves robustness to correlation shift by a compositional approach based on combining elements of images. Based on our observations, models trained with ERM usually highly attend to either the causal components or the components having a high spurious correlation with the label (especially in datapoints on which models have a high confidence). In fact, according to the amount of spurious correlation and the easiness of classification based on the causal or non-causal components, the model usually attends to one of these more (on samples with high confidence). Following this, we first try to identify the causal components of images using class activation maps of models trained with ERM. Afterwards, we intervene on images by combining them and retraining the model on the augmented data, including the counterfac-tual ones. This work proposes a group-balancing method by intervening on images without requiring group labels or information regarding the spurious features during training. The method has an overall better worst group accuracy compared to previous methods with the same amount of supervision on the group labels in correlation shift. Our code is available at https://github.com/fhn98/DaC. © 2024 IEEE
  6. Keywords:
  7. HTTP ; Image correlation ; Image enhancement ; Datapoints ; Empirical risk minimization ; High confidence ; Image components ; Images classification ; Input distributions ; Main objects ; Observation model ; Spurious correlation ; Test Environment ; Image classification
  8. Source: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition ; 2024 , Pages 27652-27661 ; 10636919 (ISSN)
  9. URL: https://ieeexplore.ieee.org/document/10657638