Login (DCU Staff Only)
Login (DCU Staff Only)

DORAS | DCU Research Repository

Explore open access research and scholarly works from DCU

Advanced Search

Two reproductions of a human-assessed comparative evaluation of a semantic error detection system

Huidrom, Rudali orcid logoORCID: 0000-0003-0630-3603, Dušek, Ondřej orcid logoORCID: 0000-0002-1415-1702, Kasner, Zdeněk orcid logoORCID: 0000-0002-5753-5538, Castro Ferrera, Thiago orcid logoORCID: 0000-0003-0200-3646 and Belz, Anya orcid logoORCID: 0000-0002-0552-8096 (2022) Two reproductions of a human-assessed comparative evaluation of a semantic error detection system. In: International Natural Language Generation Conference, 18-22 July 2022, Waterville, Maine, USA + Online.

Abstract
In this paper, we present the results of two re- production studies for the human evaluation originally reported by Dušek and Kasner (2020) in which the authors comparatively evaluated outputs produced by a semantic error detection system for data-to-text generation against ref- erence outputs. In the first reproduction, the original evaluators repeat the evaluation, in a test of the repeatability of the original evaluation. In the second study, two new evaluators carry out the evaluation task, in a test of the reproducibility of the original evaluation under otherwise identical conditions. We describe our approach to reproduction, and present and analyse results, finding different degrees of re- producibility depending on result type, data and labelling task. Our resources are available and open-sourced.
Metadata
Item Type:Conference or Workshop Item (Paper)
Event Type:Conference
Refereed:Yes
Subjects:Computer Science > Computational linguistics
DCU Faculties and Centres:DCU Faculties and Schools > Faculty of Engineering and Computing > School of Computing
Research Initiatives and Centres > ADAPT
Published in: Shaikh, Samira, Ferreira, Thiago and Stent, Amanda, (eds.) Proceedings of the 15th International Conference on Natural Language Generation: Generation Challenges. . Association for Computational Linguistics (ACL).
Publisher:Association for Computational Linguistics (ACL)
Official URL:https://aclanthology.org/2022.inlg-genchal.9
Copyright Information:© 2022 ACL
Funders:Faculty of Engineering and Computing, Dublin City University, EPSRC Grant No .EP/V05645X/1 for the Repro Hum project., ERC Grant No.101039303 NG-NLG, Czech Ministry of Education project No. LM2018101LINDAT/CLARIAH-CZ, CharlesUniversityprojectsGAUK140320andSVV260575
ID Code:28650
Deposited On:05 Jul 2023 09:46 by Anya Belz . Last Modified 05 Jul 2023 09:46
Documents

Full text available as:

[thumbnail of 2022.inlg-genchal.9.pdf]
Preview
PDF - Requires a PDF viewer such as GSview, Xpdf or Adobe Acrobat Reader
Creative Commons: Attribution 4.0
179kB
Downloads

Downloads

Downloads per month over past year

Archive Staff Only: edit this record