Abstract
Multi-modal learning focuses on training models by equally combining multiple
input data modalities during the prediction process. However, this equal
combination can be detrimental to the prediction accuracy because different
modalities are usually accompanied by varying levels of uncertainty. Using such
uncertainty to combine modalities has been studied by a couple of approaches,
but with limited success because these approaches are either designed to deal
with specific classification or segmentation problems and cannot be easily
translated into other tasks, or suffer from numerical instabilities. In this
paper, we propose a new Uncertainty-aware Multi-modal Learner that estimates
uncertainty by measuring feature density via Cross-modal Random Network
Prediction (CRNP). CRNP is designed to require little adaptation to translate
between different prediction tasks, while having a stable training process.
From a technical point of view, CRNP is the first approach to explore random
network prediction to estimate uncertainty and to combine multi-modal data.
Experiments on two 3D multi-modal medical image segmentation tasks and three 2D
multi-modal computer vision classification tasks show the effectiveness,
adaptability and robustness of CRNP. Also, we provide an extensive discussion
on different fusion functions and visualization to validate the proposed model.