Loading...

OUT-OF-DOMAIN UNLABELED DATA IMPROVES GENERALIZATION

Saberi, A. H ; Sharif University of Technology | 2024

260 Viewed
  1. Type of Document: Article
  2. Publisher: 2024
  3. Abstract:
  4. We propose a novel framework for incorporating unlabeled data into semi-supervised classification problems, where scenarios involving the minimization of either i) adversarially robust or ii) non-robust loss functions have been considered. Notably, we allow the unlabeled samples to deviate slightly (in total variation sense) from the in-domain distribution. The core idea behind our framework is to combine Distributionally Robust Optimization (DRO) with self-supervised training. As a result, we also leverage efficient polynomial-time algorithms for the training stage. From a theoretical standpoint, we apply our framework on the classification problem of a mixture of two Gaussians in Rd, where in addition to the m independent and labeled samples from the true distribution, a set of n (usually with n ≫ m) out of domain and unlabeled samples are given as well. Using only the labeled data, it is known that the generalization error can be bounded by ∝ (d/m)1/2. However, using our method on both isotropic and non-isotropic Gaussian mixture models, one can derive a new set of analytically explicit and non-asymptotic bounds which show substantial improvement on the generalization error compared to ERM. Our results underscore two significant insights: 1) out-of-domain samples, even when unlabeled, can be harnessed to narrow the generalization gap, provided that the true data distribution adheres to a form of the “cluster assumption”, and 2) the semi-supervised learning paradigm can be regarded as a special case of our framework when there are no distributional shifts. We validate our claims through experiments conducted on a variety of synthetic and real-world datasets. © 2024 12th International Conference on Learning Representations, ICLR 2024. All rights reserved
  5. Keywords:
  6. Optimization ; Supervised learning ; Domain distribution ; Generalisation ; Generalization Error ; Isotropics ; Loss functions ; Minimisation ; Semisupervised classification (SSC) ; Total-variation ; Unlabeled data ; Unlabeled samples ; Polynomial approximation
  7. Source: 12th International Conference on Learning Representations, ICLR 2024 ; 2024
  8. URL: https://dl.acm.org/doi/10.1145/3746252.3760942