Login (DCU Staff Only)
Login (DCU Staff Only)

DORAS | DCU Research Repository

Explore open access research and scholarly works from DCU

Advanced Search

A survey of recent error annotation schemes for automatically generated text

Huidrom, Rudali orcid logoORCID: 0000-0003-0630-3603 and Belz, Anya orcid logoORCID: 0000-0002-0552-8096 (2022) A survey of recent error annotation schemes for automatically generated text. In: 2nd Workshop on Natural Language Generation, Evaluation, and Metrics (GEM), 7 Dec 2022, Abu Dhabi, UAE and Online.

Abstract
While automatically computing numerical scores remains the dominant paradigm in NLP system evaluation, error analysis is receiving increasing attention, with numerous error annotation schemes being proposed for automatically generated text. However, there is little agreement about what error annotation schemes should look like, how many different types of errors should be distinguished and at what level of granularity. In this paper, our aim is to map out recent work on annotating errors in automatically generated text, with a particular focus on error taxonomies. We describe our systematic paper selection process, and survey the error annotation schemes reported in the papers, drawing out similarities and differences between them. Finally, we characterise the issues that would make it difficult to move from the current situation to a standardised error taxonomy for annotating errors in automatically generated text.
Metadata
Item Type:Conference or Workshop Item (Paper)
Event Type:Workshop
Refereed:Yes
Subjects:Computer Science > Computational linguistics
Computer Science > Machine learning
DCU Faculties and Centres:DCU Faculties and Schools > Faculty of Engineering and Computing > School of Computing
Research Initiatives and Centres > ADAPT
Published in: Proceedings of the 2nd Workshop on Natural Language Generation, Evaluation, and Metrics (GEM). . Association for Computational Linguistics (ACL).
Publisher:Association for Computational Linguistics (ACL)
Official URL:https://aclanthology.org/2022.gem-1.33.pdf
Copyright Information:© 2022 Association for Computational Linguistics
Funders:ADAPT Centre for Digital Media Technology funded by Science Foundation Ireland SFI Research Centres Programme and is co-funded under the European Regional Development Fund (ERDF) Grant 13/RC/2106.
ID Code:28660
Deposited On:04 Jul 2023 14:51 by Anya Belz . Last Modified 12 Jul 2023 11:12
Documents

Full text available as:

[thumbnail of 2022.gem-1.33.pdf]
Preview
PDF - Requires a PDF viewer such as GSview, Xpdf or Adobe Acrobat Reader
Creative Commons: Attribution 4.0
3MB
Downloads

Downloads

Downloads per month over past year

Archive Staff Only: edit this record