Login (DCU Staff Only)
Login (DCU Staff Only)

DORAS | DCU Research Repository

Explore open access research and scholarly works from DCU

Advanced Search

A metrological perspective on reproducibility in nlp

Belz, Anya orcid logoORCID: 0000-0002-0552-8096 (2022) A metrological perspective on reproducibility in nlp. Computational Linguistics, 48 (4). pp. 1125-1135. ISSN 0891-2017

Abstract
Reproducibility has become an increasingly debated topic in NLP and ML over recent years, but so far, no commonly accepted definitions of even basic terms or concepts have emerged. The range of different definitions proposed within NLP/ML not only do not agree with each other, they are also not aligned with standard scientific definitions. This article examines the standard definitions of repeatability and reproducibility provided by the meta-science of metrology, and explores what they imply in terms of how to assess reproducibility, and what adopting them would mean for reproducibility assessment in NLP/ML. It turns out the standard definitions lead directly to a method for assessing reproducibility in quantified terms that renders results from reproduction studies comparable across multiple reproductions of the same original study, as well as reproductions of different original studies. The article considers where this method sits in relation to other aspects of NLP work one might wish to assess in the context of reproducibility.
Metadata
Item Type:Article (Published)
Refereed:Yes
Subjects:Computer Science > Computational linguistics
DCU Faculties and Centres:DCU Faculties and Schools > Faculty of Engineering and Computing > School of Computing
Research Initiatives and Centres > ADAPT
Publisher:MIT Press
Official URL:https://doi.org/10.1162/coli_a_00448
Copyright Information:© 2022 Association for Computational Linguistics
ID Code:28652
Deposited On:05 Jul 2023 13:37 by Anya Belz . Last Modified 05 Jul 2023 13:37
Documents

Full text available as:

[thumbnail of coli_a_00448.pdf]
Preview
PDF - Requires a PDF viewer such as GSview, Xpdf or Adobe Acrobat Reader
Creative Commons: Attribution-Noncommercial-No Derivative Works 4.0
254kB
Downloads

Downloads

Downloads per month over past year

Archive Staff Only: edit this record