Abstract
Image anomaly detection (IAD) is an emerging and vital computer vision task
in industrial manufacturing (IM). Recently many advanced algorithms have been
published, but their performance deviates greatly. We realize that the lack of
actual IM settings most probably hinders the development and usage of these
methods in real-world applications. As far as we know, IAD methods are not
evaluated systematically. As a result, this makes it difficult for researchers
to analyze them because they are designed for different or special cases. To
solve this problem, we first propose a uniform IM setting to assess how well
these algorithms perform, which includes several aspects, i.e., various levels
of supervision (unsupervised vs. semi-supervised), few-shot learning, continual
learning, noisy labels, memory usage, and inference speed. Moreover, we
skillfully build a comprehensive image anomaly detection benchmark (IM-IAD)
that includes 16 algorithms on 7 mainstream datasets with uniform settings. Our
extensive experiments (17,017 in total) provide in-depth insights for IAD
algorithm redesign or selection under the IM setting. Next, the proposed
benchmark IM-IAD gives challenges as well as directions for the future. To
foster reproducibility and accessibility, the source code of IM-IAD is uploaded
on the website, https://github.com/M-3LAB/IM-IAD.