2023

Belz, Anya ORCID: 0000-0002-0552-8096, Thomson, Craig, Reiter, Ehud ORCID: 0000-0002-7548-9504, Abercrombie, Gavin ORCID: 0000-0002-6546-3562, Alonso-Moral, Jose M. ORCID: 0000-0003-3673-421X, Arvan, Mohammad, Cheung, Jackie, Cieliebak, Mark ORCID: 0009-0007-3059-8516, Clark, Elizabeth, van Deemter, Kees, Kelleher, John D. ORCID: 0000-0001-6462-3248 and Klubička, Filip ORCID: 0000-0001-9712-6141 (2023) Missing information, unresponsive authors, experimental flaws: the impossibility of assessing the reproducibility of previous human evaluations in NLP. In: Fourth Workshop on Insights from Negative Results in NLP, 2 May 2023, Dubrovnik, Croatia. ISBN 978-1-959429-49-4

2022

Savkov, Aleksandar ORCID: 0009-0009-6831-5563, Moramarco, Francesco, Korfiatis, Alex Papadopoulos, Perera, Mark, Belz, Anya ORCID: 0000-0002-0552-8096 and Reiter, Ehud ORCID: 0000-0002-7548-9504 (2022) Consultation checklists: standardising the human evaluation of medical note generation. In: EMNLP 2022 Industry Track, 9-11 Dec 2022, Abu Dhabi, UAE.

Belz, Anya ORCID: 0000-0002-0552-8096, Shimorina, Anastasia, Popović, Maja ORCID: 0000-0001-8234-8745 and Reiter, Ehud ORCID: 0000-0002-7548-9504 (2022) The 2022 ReproGen shared task on reproducibility of evaluations in NLG: overview and results. In: 15th International Conference on Natural Language Generation: Generation Challenges, 17-22 July 2022, Waterville, ME, USA.

Knoll, Tom, Moramarco, Francesco, Korfiatis, Alex Papadopoulos, Young, Rachel, Ruffini, Claudia, Perera, Mark, Perstl, Christian, Reiter, Ehud, Belz, Anya ORCID: 0000-0002-0552-8096 and Savkov, Aleksandar ORCID: 0009-0009-6831-5563 (2022) User-driven development of a medical note generation system. In: 20th Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL'22), 10-15 July 2022, Seattle, Washington, USA.

Shimorina, Anastasia and Belz, Anya ORCID: 0000-0002-0552-8096 (2022) The human evaluation datasheet: a template for recording details of human evaluation experiments in NLP. In: 2nd Workshop on Human Evaluation of NLP Systems, 27 May 2022, Dublin, Ireland. ISBN 9781713867395

Moramarco, Francesco, Korfiatis, Alex Papadopoulos, Perera, Mark, Juric, Damir, Flann, Jack, Reiter, Ehud, Belz, Anya ORCID: 0000-0002-0552-8096 and Savkov, Aleksandar ORCID: 0009-0009-6831-5563 (2022) Human evaluation and correlation with automatic metrics in consultation note generation. In: 60th Annual Meeting of the Association for Computational Linguistics, 22-27 May 2022, Dublin, Ireland.

2021

Mille, Simon ORCID: 0000-0002-8852-2764, Castro Ferreira, Thiago ORCID: 0000-0003-0200-3646, Davis, Brian ORCID: 0000-0002-5759-2655 and Belz, Anya ORCID: 0000-0002-0552-8096 (2021) Another PASS: a reproduction study of the human evaluation of a football report generation system. In: 14th International Conference on Natural Language Generation (INLG 2021), 22-27 May 2022, Aberdeen, Scotland.

Belz, Anya ORCID: 0000-0002-0552-8096, Agarwal, Shubham, Shimorina, Anastasia and Reiter, Ehud (2021) A systematic review of reproducibility research in natural language processing. In: 16th Conference of the European Chapter of the Association for Computational Linguistics: EACL'21, 19 - 23 Apr 2021, Online. ISBN 978-1-954085-02-2