Login (DCU Staff Only)
Login (DCU Staff Only)

DORAS | DCU Research Repository

Explore open access research and scholarly works from DCU

Advanced Search

cushLEPOR uses LABSE distilled knowledge to improve correlation with human translation evaluations

Erofeev, Gleb, Sorokina, Irina, Han, Lifeng orcid logoORCID: 0000-0002-3221-2185 and Gladkoff, Serge (2021) cushLEPOR uses LABSE distilled knowledge to improve correlation with human translation evaluations. In: Machine Translation Summit 2021, 16-20 Aug 2021, USA (online).

Abstract
Human evaluation has always been expensive while researchers struggle to trust the automatic metrics. To address this, we propose to customise traditional metrics by taking advantages of the pre-trained language models (PLMs) and the limited available human labelled scores. We first re-introduce the hLEPOR metric factors, followed by the Python portable version we developed which achieved the automatic tuning of the weighting parameters in hLEPOR metric. Then we present the customised hLEPOR (cushLEPOR) which uses LABSE distilled knowledge model to improve the metric agreement with human judgements by automatically optimised factor weights regarding the exact MT language pairs that cushLEPOR is deployed to. We also optimise cushLEPOR towards human evaluation data based on MQM and pSQM framework on English-German and Chinese-English language pairs. The experimental investigations show cushLEPOR boosts hLEPOR performances towards better agreements to PLMs like LABSE with much lower cost, and better agreements to human evaluations including MQM and pSQM scores, and yields much better performances than BLEU (data available at \url{this https URL}).
Metadata
Item Type:Conference or Workshop Item (Speech)
Event Type:Conference
Refereed:Yes
Uncontrolled Keywords:Machine Translation Evaluation; Parameter Optimisation; Evaluation Metrics; Agreement; Statistical Analysis
Subjects:Computer Science > Algorithms
Computer Science > Artificial intelligence
Computer Science > Computational linguistics
Computer Science > Computer engineering
Computer Science > Information technology
Computer Science > Machine learning
Mathematics > Mathematical models
Mathematics > Statistics
DCU Faculties and Centres:DCU Faculties and Schools > Faculty of Engineering and Computing > School of Computing
Research Initiatives and Centres > ADAPT
Published in: Proceedings of Machine Translation Summit XVIII: Users and Providers Track. . Association for Computational Linguistics.
Publisher:Association for Computational Linguistics
Official URL:https://aclanthology.org/2021.mtsummit-up.28
Copyright Information:© 2021 The Authors
Use License:This item is licensed under a Creative Commons Attribution-NonCommercial-Share Alike 3.0 License. View License
Funders:ADAPT
ID Code:26182
Deposited On:07 Sep 2021 13:04 by Lifeng Han . Last Modified 16 Jan 2023 16:14
Documents

Full text available as:

[thumbnail of cushLEPOR-MTsummit-2021_final.pdf]
Preview
PDF - Requires a PDF viewer such as GSview, Xpdf or Adobe Acrobat Reader
Creative Commons: Attribution-Noncommercial 3.0
901kB
Downloads

Downloads

Downloads per month over past year

Archive Staff Only: edit this record