Abstract
Understanding dynamic multimedia content requires models to continually learn new visual concepts while preserving previously acquired scene knowledge. In continual panoptic segmentation (CPS), this challenge goes beyond category recognition, since robust performance also depends on maintaining coherent semantic relations, spatial layouts, and instance-level distinctions over time. Existing CPS methods mainly rely on evolving visual features as the reference for knowledge preservation, which becomes increasingly fragile under long incremental learning. We propose Language-Anchored Hierarchical Relational Distillation (LA-HRD), a framework for continual multimodal scene understanding. LA-HRD stabilizes scene representations at three levels: frozen language embeddings provide a semantic anchor for preserving global class topology; visual distillation maintains spatial-contextual structure; and prototype-guided contrastive alignment improves instance-level compactness and temporal consistency. By aligning different forms of knowledge in their appropriate reference spaces, LA-HRD enables more robust long-term scene parsing. Experiments on multiple ADE20K continual protocols and additional COCO evaluation show that LA-HRD consistently improves the stability-plasticity trade-off and produces more coherent scene parsing results over long incremental sequences.