Login (DCU Staff Only)
Login (DCU Staff Only)

DORAS | DCU Research Repository

Explore open access research and scholarly works from DCU

Advanced Search

New metrics for meaningful evaluation of informally structured speech retrieval

Eskevich, Maria orcid logoORCID: 0000-0002-1242-0753, Magdy, Walid and Jones, Gareth J.F. orcid logoORCID: 0000-0003-2923-8365 (2012) New metrics for meaningful evaluation of informally structured speech retrieval. In: 34th European Conference on Information Retrieval Research,, 1-5 Apr 2012, Barcelona, Spain. ISBN 978-3-642-28997-2

Abstract
Search effectiveness for tasks where the retrieval units are clearly defined documents is generally evaluated using standard measures such as mean average precision (MAP). However, many practical speech search tasks focus on content within large spoken files lacking defined structure. These data must be segmented into smaller units for search which may only partially overlap with relevant material. We introduce two new metrics for the evaluation of search effectiveness for informally structured speech data: mean average segment precision (MASP) which measures retrieval performance in terms of both content segmentation and ranking with respect to relevance; and mean average segment distance-weighted precision (MASDWP) which takes into account the distance between the start of the relevant segment and the retrieved segment. We demonstrate the eectiveness of these new metrics on a retrieval test collection based on the AMI meeting corpus.
Metadata
Item Type:Conference or Workshop Item (Paper)
Event Type:Conference
Refereed:Yes
Uncontrolled Keywords:Speech retrieval; informally structured speech; evaluation metrics
Subjects:Computer Science > Multimedia systems
Computer Science > Information retrieval
DCU Faculties and Centres:Research Initiatives and Centres > Centre for Digital Video Processing (CDVP)
Research Initiatives and Centres > Centre for Next Generation Localisation (CNGL)
DCU Faculties and Schools > Faculty of Engineering and Computing > School of Computing
Published in: Advances in Information Retrieval. Lecture Notes in Computer Science (LNCS) 7224. Springer. ISBN 978-3-642-28997-2
Publisher:Springer
Official URL:http://dx.doi.org/10.1007/978-3-642-28997-2_15
Copyright Information:© 2012 Springer-Verlag. The original publication is available at www.springerlink.com
Use License:This item is licensed under a Creative Commons Attribution-NonCommercial-Share Alike 3.0 License. View License
Funders:Science Foundation Ireland
ID Code:16900
Deposited On:23 Apr 2012 11:04 by Gareth Jones . Last Modified 10 Oct 2018 09:20
Documents

Full text available as:

[thumbnail of full paper from ECIR 2012]
Preview
PDF (full paper from ECIR 2012) - Requires a PDF viewer such as GSview, Xpdf or Adobe Acrobat Reader
254kB
Downloads

Downloads

Downloads per month over past year

Archive Staff Only: edit this record