Login (DCU Staff Only)
Login (DCU Staff Only)

DORAS | DCU Research Repository

Explore open access research and scholarly works from DCU

Advanced Search

On the same page? Comparing inter-annotator agreement in sentence and document level human machine translation evaluation

Castilho, Sheila orcid logoORCID: 0000-0002-8416-6555 (2020) On the same page? Comparing inter-annotator agreement in sentence and document level human machine translation evaluation. In: Fifth Conference on Machine Translation, 19-20 Nov 2020, Dominican Republic (Online).

Abstract
Document-level evaluation of machine translation has raised interest in the community especially since responses to the claims of “human parity” (Toral et al., 2018; L¨aubli et al.,2018) with document-level human evaluations have been published. Yet, little is known about best practices regarding human evaluation of machine translation at the documentlevel. This paper presents a comparison of the differences in inter-annotator agreement between quality assessments using sentence and document-level set-ups. We report results of the agreement between professional translators for fluency and adequacy scales, error annotation, and pair-wise ranking, along with the effort needed to perform the different tasks. To best of our knowledge, this is the first study of its kind.
Metadata
Item Type:Conference or Workshop Item (Paper)
Event Type:Conference
Refereed:Yes
Subjects:Computer Science > Machine translating
Humanities > Translating and interpreting
DCU Faculties and Centres:DCU Faculties and Schools > Faculty of Engineering and Computing > School of Computing
Research Initiatives and Centres > ADAPT
Published in: Proceedings of the Fifth Conference on Machine Translation. . Association for Computational Linguistics (ACL).
Publisher:Association for Computational Linguistics (ACL)
Official URL:https://www.aclweb.org/anthology/2020.wmt-1.137
Copyright Information:© 2020 The Author. CC-BY- 4.0
Funders:European Association for Machine Translation, Science Foundation Ireland Research Centres Programme (Grant 13/RC/2106) and is co-funded by the European Regional Development Fund.
ID Code:25075
Deposited On:12 Oct 2020 14:30 by Sheila Castilho . Last Modified 12 Jan 2021 12:13
Documents

Full text available as:

[thumbnail of EMNLP_2020_Document agreement.pdf]
Preview
PDF - Requires a PDF viewer such as GSview, Xpdf or Adobe Acrobat Reader
266kB
Downloads

Downloads

Downloads per month over past year

Archive Staff Only: edit this record