Login (DCU Staff Only)
Login (DCU Staff Only)

DORAS | DCU Research Repository

Explore open access research and scholarly works from DCU

Advanced Search

Human evaluation and correlation with automatic metrics in consultation note generation

Moramarco, Francesco, Korfiatis, Alex Papadopoulos, Perera, Mark, Juric, Damir, Flann, Jack, Reiter, Ehud, Belz, Anya orcid logoORCID: 0000-0002-0552-8096 and Savkov, Aleksandar orcid logoORCID: 0009-0009-6831-5563 (2022) Human evaluation and correlation with automatic metrics in consultation note generation. In: 60th Annual Meeting of the Association for Computational Linguistics, 22-27 May 2022, Dublin, Ireland.

Abstract
In recent years, machine learning models have rapidly become better at generating clinical consultation notes; yet, there is little work on how to properly evaluate the generated consultation notes to understand the impact they may have on both the clinician using them and the patient’s clinical safety.To address this we present an extensive human evaluation study of consultation notes where 5 clinicians (i) listen to 57 mock consultations, (ii) write their own notes, (iii) post-edit a number of automatically generated notes, and (iv) extract all the errors, both quantitative and qualitative. We then carry out a correlation study with 18 automatic quality metrics and the human judgements. We find that a simple, character-based Levenshtein distance metric performs on par if not better than common model-based metrics like BertScore. All our findings and annotations are open-sourced.
Metadata
Item Type:Conference or Workshop Item (Paper)
Event Type:Conference
Refereed:Yes
Subjects:Computer Science > Computational linguistics
DCU Faculties and Centres:DCU Faculties and Schools > Faculty of Engineering and Computing > School of Computing
Research Initiatives and Centres > ADAPT
Published in: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics. 1. Association for Computational Linguistics (ACL).
Publisher:Association for Computational Linguistics (ACL)
Official URL:https://doi.org/10.18653/v1/2022.acl-long.394
Copyright Information:© 2022 ACL
ID Code:28644
Deposited On:06 Jul 2023 10:14 by Anya Belz . Last Modified 06 Jul 2023 12:15
Documents

Full text available as:

[thumbnail of 2022.acl-long.394.pdf]
Preview
PDF - Requires a PDF viewer such as GSview, Xpdf or Adobe Acrobat Reader
Creative Commons: Attribution 4.0
2MB
Downloads

Downloads

Downloads per month over past year

Archive Staff Only: edit this record