Ozge Mercanoglu Sincan

Sign Language Translation (SLT) is a challenging task that aims to generate
spoken language sentences from sign language videos, both of which have
different grammar and word/gloss order. From a Neural Machine Translation (NMT)
perspective, the straightforward way of training translation models is to use
sign language phrase-spoken language sentence pairs. However, human
interpreters heavily rely on the context to understand the conveyed
information, especially for sign language interpretation, where the vocabulary
size may be significantly smaller than their spoken language equivalent.
Taking direct inspiration from how humans translate, we propose a novel
multi-modal transformer architecture that tackles the translation task in a
context-aware manner, as a human would. We use the context from previous
sequences and confident predictions to disambiguate weaker visual cues. To
achieve this we use complementary transformer encoders, namely: (1) A Video
Encoder, that captures the low-level video features at the frame-level, (2) A
Spotting Encoder, that models the recognized sign glosses in the video, and (3)
A Context Encoder, which captures the context of the preceding sign sequences.
We combine the information coming from these encoders in a final transformer
decoder to generate spoken language translations.
We evaluate our approach on the recently published large-scale BOBSL dataset,
which contains ~1.2M sequences, and on the SRF dataset, which was part of the
WMT-SLT 2022 challenge. We report significant improvements on state-of-the-art
translation performance using contextual information, nearly doubling the
reported BLEU-4 scores of baseline approaches.

Capturing and annotating Sign language datasets is a time consuming and
costly process. Current datasets are orders of magnitude too small to
successfully train unconstrained \acf{slt} models. As a result, research has
turned to TV broadcast content as a source of large-scale training data,
consisting of both the sign language interpreter and the associated audio
subtitle. However, lack of sign language annotation limits the usability of
this data and has led to the development of automatic annotation techniques
such as sign spotting. These spottings are aligned to the video rather than the
subtitle, which often results in a misalignment between the subtitle and
spotted signs. In this paper we propose a method for aligning spottings with
their corresponding subtitles using large spoken language models. Using a
single modality means our method is computationally inexpensive and can be
utilized in conjunction with existing alignment techniques. We quantitatively
demonstrate the effectiveness of our method on the \acf{mdgs} and \acf{bobsl}
datasets, recovering up to a 33.22 BLEU-1 score in word alignment.