Abstract
In a scenario where multi-modal cameras are operating together, the problem
of working with non-aligned images cannot be avoided. Yet, existing image
fusion algorithms rely heavily on strictly registered input image pairs to
produce more precise fusion results, as a way to improve the performance of
downstream high-level vision tasks. In order to relax this assumption, one can
attempt to register images first. However, the existing methods for registering
multiple modalities have limitations, such as complex structures and reliance
on significant semantic information. This paper aims to address the problem of
image registration and fusion in a single framework, called BusRef. We focus on
Infrared-Visible image registration and fusion task (IVRF). In this framework,
the input unaligned image pairs will pass through three stages: Coarse
registration, Fine registration and Fusion. It will be shown that the unified
approach enables more robust IVRF. We also propose a novel training and
evaluation strategy, involving the use of masks to reduce the influence of
non-reconstructible regions on the loss functions, which greatly improves the
accuracy and robustness of the fusion task. Last but not least, a
gradient-aware fusion network is designed to preserve the complementary
information. The advanced performance of this algorithm is demonstrated by