Logo image
Benchmarking terminology building capabilities of ChatGPT on an English-Russian Fashion Corpus
Conference proceeding   Open access

Benchmarking terminology building capabilities of ChatGPT on an English-Russian Fashion Corpus

Anastasiia Bezobrazova, Constantin Orasan and Miriam Seghiri
Translating and the Computer 2024 (TC46) proceedings
Translating and the Computer 2024 , 46 (European Convention Center, Luxembourg, 18/11/2024–20/11/2024)
03/2025

Abstract

This paper compares the accuracy of the terms extracted using SketchEngine, TBXTools and ChatGPT. In addition, it evaluates the quality of the definitions produced by ChatGPT for these terms. The research is carried out on a comparable corpus of fashion magazines written in English and Russian collected from the web. A gold standard for the fashion terminology was also developed by identifying web pages that can be harvested automatically and contain definitions of terms from the fashion domain in English and Russian. This gold standard was used to evaluate the quality of the extracted terms and of the definitions produced. Our evaluation shows that TBXTools and SketchEngine, while capable of high recall, suffer from reduced precision as the number of terms increases, which affects their overall performance. Conversely, ChatGPT demonstrates superior performance, maintaining or improving precision as more terms are considered. Analysis of the definitions produced by ChatGPT for 60 commonly used terms in English and Russian shows that ChatGPT maintains a reasonable level of accuracy and fidelity across languages, but sometimes the definitions in both languages miss crucial specifics and include unnecessary deviations. Our research reveals that no single tool excels universally; each has strengths suited to particular aspects of terminology extraction and application.
pdf
2412.03242v1637.51 kBDownloadView
Open Access
url
https://asling.org/tc46/View
Event Website Conference website
url
https://asling.org/tc46/wp-content/uploads/2025/03/TC46-proceedings.pdfView
Conference proceedings

Metrics

2 File views/ downloads
4 Record Views

Details

Logo image

Usage Policy