Login (DCU Staff Only)
Login (DCU Staff Only)

DORAS | DCU Research Repository

Explore open access research and scholarly works from DCU

Advanced Search

Reproducing a manual evaluation of the simplicity of text simplification system outputs

Popović, Maja orcid logoORCID: 0000-0001-8234-8745, Castilho, Sheila orcid logoORCID: 0000-0002-8416-6555, Huidrom, Rudali orcid logoORCID: 0000-0003-0630-3603 and Belz, Anya orcid logoORCID: 0000-0002-0552-8096 (2022) Reproducing a manual evaluation of the simplicity of text simplification system outputs. In: 15th International Conference on Natural Language Generation: Generation Challenges, 17-22 July 2022, Waterville, ME, USA.

Abstract
In this paper we describe our reproduction study of the human evaluation of text simplic- ity reported by Nisioi et al. (2017). The work was carried out as part of the ReproGen Shared Task 2022 on Reproducibility of Evaluations in NLG. Our aim was to repeat the evaluation of simplicity for nine automatic text simplification systems with a different set of evaluators. We describe our experimental design together with the known aspects of the original experimental design and present the results from both studies. Pearson correlation between the original and reproduction scores is moderate to high (0.776). Inter-annotator agreement in the reproduction study is lower (0.40) than in the original study (0.66). We discuss challenges arising from the unavailability of certain aspects of the origi- nal set-up, and make several suggestions as to how reproduction of similar evaluations can be made easier in future.
Metadata
Item Type:Conference or Workshop Item (Paper)
Event Type:Conference
Refereed:Yes
Subjects:Computer Science > Computational linguistics
DCU Faculties and Centres:DCU Faculties and Schools > Faculty of Engineering and Computing > School of Computing
Research Initiatives and Centres > ADAPT
Published in: Proceedings of the 15th International Conference on Natural Language Generation: Generation Challenges. . Association for Computational Linguistics (ACL).
Publisher:Association for Computational Linguistics (ACL)
Official URL:https://aclanthology.org/2022.inlg-genchal.12
Copyright Information:©2022 Association for Computational Linguistics
Funders:Science Foundation Ireland through the SFI Research Centres Programme and is co-funded under the European Regional Development Fund (ERDF) through Grant 13/RC/2106, Faculty of Engineering and Computing, Dublin City University
ID Code:28368
Deposited On:25 May 2023 14:21 by Maja Popovic . Last Modified 04 Jul 2023 10:39
Documents

Full text available as:

[thumbnail of 2022.inlg-genchal.12.pdf]
Preview
PDF - Requires a PDF viewer such as GSview, Xpdf or Adobe Acrobat Reader
Creative Commons: Attribution 4.0
123kB
Downloads

Downloads

Downloads per month over past year

Archive Staff Only: edit this record