Login (DCU Staff Only)
Login (DCU Staff Only)

DORAS | DCU Research Repository

Explore open access research and scholarly works from DCU

Advanced Search

PRES: A score metric for evaluating recall-oriented information retrieval applications

Magdy, Walid and Jones, Gareth J.F. orcid logoORCID: 0000-0003-2923-8365 (2010) PRES: A score metric for evaluating recall-oriented information retrieval applications. In: The 33rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2010), 19-23 Jul 2010, Geneva, Switzerland. ISBN 978-1-4503-0153-4

Abstract
Information retrieval (IR) evaluation scores are generally designed to measure the effectiveness with which relevant documents are identified and retrieved. Many scores have been proposed for this purpose over the years. These have primarily focused on aspects of precision and recall, and while these are often discussed with equal importance, in practice most attention has been given to precision focused metrics. Even for recalloriented IR tasks of growing importance, such as patent retrieval, these precision based scores remain the primary evaluation measures. Our study examines different evaluation measures for a recall-oriented patent retrieval task and demonstrates the limitations of the current scores in comparing different IR systems for this task. We introduce PRES, a novel evaluation metric for this type of application taking account of recall and the user’s search effort. The behaviour of PRES is demonstrated on 48 runs from the CLEF-IP 2009 patent retrieval track. A full analysis of the performance of PRES shows its suitability for measuring the retrieval effectiveness of systems from a recall focused perspective taking into account the user’s expected search effort.
Metadata
Item Type:Conference or Workshop Item (Paper)
Event Type:Conference
Refereed:Yes
Uncontrolled Keywords:PRES; Recall-Oriented Information Retrieval; Patent Retrieval; Evaluation Metric
Subjects:Computer Science > Information retrieval
DCU Faculties and Centres:Research Initiatives and Centres > Centre for Next Generation Localisation (CNGL)
DCU Faculties and Schools > Faculty of Engineering and Computing > School of Computing
Published in: SIGIR 2010 Proceedings - 33rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. . Association for Computing Machinery. ISBN 978-1-4503-0153-4
Publisher:Association for Computing Machinery
Official URL:http://portal.acm.org/ft_gateway.cfm?id=1835551&ty...
Copyright Information:© ACM, 2010. This is the author's version of the work. It is posted here by permission of ACM for your personal use. Not for redistribution. The definitive version was published in SIGIR 2010 Proceedings - 33rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, 2010 http://doi.acm.org/10.1145/1835449.1835551
Use License:This item is licensed under a Creative Commons Attribution-NonCommercial-Share Alike 3.0 License. View License
ID Code:16180
Deposited On:22 Jul 2011 10:55 by Shane Harper . Last Modified 25 Oct 2018 10:49
Documents

Full text available as:

[thumbnail of PRES_A_Score_Metric_for_Evaluating_Recall-Oriented_Information_Retrieval_Applications.pdf]
Preview
PDF - Requires a PDF viewer such as GSview, Xpdf or Adobe Acrobat Reader
661kB
Downloads

Downloads

Downloads per month over past year

Archive Staff Only: edit this record