Abstract
Semi-supervised learning (SSL) aims to avoid the need for collecting prohibitively expensive labelled training data. Whilst
demonstrating impressive performance boost, existing SSL
methods artificially assume that small labelled data and large
unlabelled data are drawn from the same class distribution. In
a more realistic scenario with class distribution mismatch between the two sets, they often suffer severe performance degradation due to error propagation introduced by irrelevant unlabelled samples. Our work addresses this under-studied and realistic SSL problem by a novel algorithm named UncertaintyAware Self-Distillation (UASD). Specifically, UASD produces
soft targets that avoid catastrophic error propagation, and empower learning effectively from unconstrained unlabelled data
with out-of-distribution (OOD) samples. This is based on joint
Self-Distillation and OOD filtering in a unified formulation.
Without bells and whistles, UASD significantly outperforms
six state-of-the-art methods in more realistic SSL under class
distribution mismatch on three popular image classification
datasets: CIFAR10, CIFAR100, and TinyImageNet.