Abstract
Multi-modal object tracking has recently gained significant attention, as auxiliary sensors (thermal, event, and depth cameras) provide complementary information beyond standard RGB sensors. Most existing trackers are independently designed to handle multi-modal tracking tasks (e.g., RGB-T, RGB-E, and RGB-D) as single-task models, making it inherently challenging to leverage the benefits of broader multi-modal training and limiting cross-modal knowledge sharing. Although some efforts try to develop a unified system capable of addressing all tasks simultaneously, these systems often require explicit task identification. To address these limitations, we propose Switcher, a unified framework that learns dynamic modality-aware selection and integration for customised multi-modal object tracking. Specifically, we design a modality-aware classifier to achieve modality-adaptive control of processing joint image pairs. Furthermore, to address the substantial heterogeneity among different sensing modalities, we introduce a customised processing strategy that aligns modality-specific representations within one single set of parameter space, allowing diverse tracking tasks to be jointly optimized within a single unified model. Extensive experiments on RGB-T, RGB-E, and RGB-D tracking benchmarks demonstrate that Switcher achieves competitive performance, while introducing only 2.10M additional training parameters and 1.95G FLOPs. The source code wiil be available at https://github.com/ouha1998/Switcher.
•A novel unified multi-modal tracker with dynamic modality awareness.•Modality customisation adapts RGB-pretrained models to diverse modality spaces.•Extensive experiments show superior performance.