Login (DCU Staff Only)
Login (DCU Staff Only)

DORAS | DCU Research Repository

Explore open access research and scholarly works from DCU

Advanced Search

Another PASS: a reproduction study of the human evaluation of a football report generation system

Mille, Simon orcid logoORCID: 0000-0002-8852-2764, Castro Ferreira, Thiago orcid logoORCID: 0000-0003-0200-3646, Davis, Brian orcid logoORCID: 0000-0002-5759-2655 and Belz, Anya orcid logoORCID: 0000-0002-0552-8096 (2021) Another PASS: a reproduction study of the human evaluation of a football report generation system. In: 14th International Conference on Natural Language Generation (INLG 2021), 22-27 May 2022, Aberdeen, Scotland.

Abstract
This paper reports results from a reproduction study in which we repeated the human evaluation of the PASS Dutch-language football report generation system (van der Lee et al., 2017). The work was carried out as part of the ReproGen Shared Task on Reproducibility of Human Evaluations in NLG, in Track A (Paper 1). We aimed to repeat the original study exactly, with the main difference that a different set of evaluators was used. We describe the study design, present the results from the original and the reproduction study, and then compare and analyse the differences between the two sets of results. For the two ‘headline’ results of average Fluency and Clarity, we find that in both studies, the system was rated more highly for Clarity than for Fluency, and Clarity had higher standard deviation. Clarity and Fluency ratings were higher, and their standard deviations lower, in the reproduction study than in the original study by substantial margins. Clarity had a higher degree of reproducibility than Fluency, as measured by the coefficient of variation. Data and code are publicly available.
Metadata
Item Type:Conference or Workshop Item (Paper)
Event Type:Conference
Refereed:Yes
Subjects:Computer Science > Computational linguistics
DCU Faculties and Centres:DCU Faculties and Schools > Faculty of Engineering and Computing > School of Computing
Research Initiatives and Centres > ADAPT
Published in: Belz, Anya, Fan, Angela, Reiter, Ehud and Sripada, Yaji, (eds.) Procceedings of the 14th International Conference on Natural Language Generation (INLG 2021). 1. Association for Computational Linguistics (ACL).
Publisher:Association for Computational Linguistics (ACL)
Official URL:https://aclanthology.org/2021.inlg-1.30
Copyright Information:© 2021 Association for Computational Linguistics
Funders:European Commission under the H2020 program contract numbers 786731, 825079, 870930 and 952133, Science Foundation Ireland through the SFI Research Centres Programme, and co-funded under the European Regional Development Fund (ERDF) Grant 13/RC/2106., Brazilian agency CAPES under Post-doctoral grant No. 88887.508597/2020-00
ID Code:28637
Deposited On:06 Jul 2023 15:16 by Anya Belz . Last Modified 10 Jul 2023 08:30
Documents

Full text available as:

[thumbnail of 2022.acl-long.2 (1).pdf]
Preview
PDF - Requires a PDF viewer such as GSview, Xpdf or Adobe Acrobat Reader
Creative Commons: Attribution 4.0
363kB
Downloads

Downloads

Downloads per month over past year

Archive Staff Only: edit this record