Login (DCU Staff Only)
Login (DCU Staff Only)

DORAS | DCU Research Repository

Explore open access research and scholarly works from DCU

Advanced Search

Head first: living labs for ad-hoc search evaluation

Balog, Krisztian, Kelly, Liadh orcid logoORCID: 0000-0003-1131-5238 and Schuth, Anne (2014) Head first: living labs for ad-hoc search evaluation. In: 23rd ACM International Conference on Information and Knowledge Management (CIKM 2014), 3-7 Nov 2014, Shanghai, China. ISBN 978-1-4503-2598-1

Abstract
The information retrieval (IR) community strives to make evaluation more centered on real users and their needs. The living labs evaluation paradigm, i.e., observing users in their natural task environments, offers great promise in this regard. Yet, progress in an academic setting has been limited. This paper presents the first living labs for the IR community benchmarking campaign initiative, taking as test two use-cases: local domain search on a university website and product search on an e-commerce site. There are many challenges associated with this setting, including incorporating results from experimental search systems into live production systems, and obtaining sufficiently many impressions from relatively low traffic sites. We propose that head queries can be used to generate result lists offline, which are then interleaved with results of the production system for live evaluation. An API is developed to orchestrate the communication between commercial parties and benchmark participants. This campaign acts to progress the living labs for IR evaluation methodology, and offers important insight into the role of living labs in this space.
Metadata
Item Type:Conference or Workshop Item (Paper)
Event Type:Conference
Refereed:Yes
Uncontrolled Keywords:Evaluation; Living Labs
Subjects:Computer Science > Information retrieval
DCU Faculties and Centres:Research Initiatives and Centres > Centre for Next Generation Localisation (CNGL)
DCU Faculties and Schools > Faculty of Engineering and Computing > School of Computing
Published in: Proceedings of the 23rd ACM International Conference on Conference on Information and Knowledge Management. . Association for Computing Machinery. ISBN 978-1-4503-2598-1
Publisher:Association for Computing Machinery
Official URL:http://dx.doi.org/10.1145/2661829.2661962
Copyright Information:© ACM, 2014. This is the author's version of the work. It is posted here by permission of ACM for your personal use. Not for redistribution. The definitive version was published in Proceedings of the 23rd ACM International Conference on Conference on Information and Knowledge Management http://dx.doi.org/10.1145/2661829.2661962
Use License:This item is licensed under a Creative Commons Attribution-NonCommercial-Share Alike 3.0 License. View License
ID Code:20131
Deposited On:26 Mar 2015 14:01 by Liadh Kelly . Last Modified 09 Nov 2018 13:49
Documents

Full text available as:

[thumbnail of cikm2014-lleval.pdf]
Preview
PDF - Requires a PDF viewer such as GSview, Xpdf or Adobe Acrobat Reader
193kB
Downloads

Downloads

Downloads per month over past year

Archive Staff Only: edit this record