Login (DCU Staff Only)
Login (DCU Staff Only)

DORAS | DCU Research Repository

Explore open access research and scholarly works from DCU

Advanced Search

A study of inter-annotator agreement for opinion retrieval

Bermingham, Adam and Smeaton, Alan F. (2009) A study of inter-annotator agreement for opinion retrieval. In: SIGIR 2009 - The 32nd Annual ACM SIGIR Conference, 20-22 July 2009, Boston, USA. ISBN 978-1-60558-483-6

Abstract
Evaluation of sentiment analysis, like large-scale IR evalu- ation, relies on the accuracy of human assessors to create judgments. Subjectivity in judgments is a problem for rel- evance assessment and even more so in the case of senti- ment annotations. In this study we examine the degree to which assessors agree upon sentence-level sentiment anno- tation. We show that inter-assessor agreement is not con- tingent on document length or frequency of sentiment but correlates positively with automated opinion retrieval per- formance. We also examine the individual annotation cate- gories to determine which categories pose most di±culty for annotators.
Metadata
Item Type:Conference or Workshop Item (Poster)
Event Type:Conference
Refereed:Yes
Subjects:Computer Science > Computational linguistics
Computer Science > Machine learning
Computer Science > Information retrieval
DCU Faculties and Centres:DCU Faculties and Schools > Faculty of Engineering and Computing > School of Computing
Research Initiatives and Centres > CLARITY: The Centre for Sensor Web Technologies
Published in: SIGIR 2009: Proceedings of the 32nd international ACM SIGIR conference on Research and development in information retrieval. . Association for Computing Machinery. ISBN 978-1-60558-483-6
Publisher:Association for Computing Machinery
Official URL:http://dx.doi.org/10.1145/1571941.1572127
Copyright Information:Copyright 2009 the authors
Use License:This item is licensed under a Creative Commons Attribution-NonCommercial-Share Alike 3.0 License. View License
Funders:Science Foundation Ireland, SFI 07/CE/I1147
ID Code:14779
Deposited On:05 Aug 2009 14:48 by Adam Bermingham . Last Modified 19 Jul 2018 14:48
Documents

Full text available as:

[thumbnail of sigir146-bermingham.pdf]
Preview
PDF - Requires a PDF viewer such as GSview, Xpdf or Adobe Acrobat Reader
207kB
Downloads

Downloads

Downloads per month over past year

Archive Staff Only: edit this record