Login (DCU Staff Only)
Login (DCU Staff Only)

DORAS | DCU Research Repository

Explore open access research and scholarly works from DCU

Advanced Search

Causality inspired retrieval of human-object interactions from video

Zhou, Liting orcid logoORCID: 0000-0002-7778-8743 and Gurrin, Cathal orcid logoORCID: 0000-0003-2903-3968 (2019) Causality inspired retrieval of human-object interactions from video. In: 2019 International Conference on Content-Based Multimedia Indexing (CBMI), 4-6 Sept 2019, Dublin, Ireland. ISBN 978-1-7281-4673-7

Abstract
Notwithstanding recent advances in machine vision, video activity recognition from multiple cameras still remains a challenging task as many real-world interactions cannot be automatically recognised for many reasons, such as partial occlusion or coverage black-spots. In this paper we propose a new technique that infers the unseen relationship between two individuals captured by different cameras and use it to retrieve relevant video clips if there is a likely interaction between the two individuals. We introduce a human object interaction (HOI) model integrating the causal relationship between the humans and the objects. For this we first extract the key frames and generate the labels or annotations using the state-of-the-art image captioning models. Next, we extract SVO (subject, verb, object) triples and encode the descriptions into a vector form for HOI inference using the Stanford CoreNLP parser. In order to calculate the HOI co-existence and the possible causality score we use transfer entropy. From our experimentation, we found that integrating casual relations into the content indexing process and using transfer entropy to calculate the causality score leads to improvement in retrieval performance.
Metadata
Item Type:Conference or Workshop Item (Paper)
Event Type:Conference
Refereed:Yes
Subjects:Computer Science > Multimedia systems
Computer Science > Digital video
Computer Science > Lifelog
DCU Faculties and Centres:DCU Faculties and Schools > Faculty of Engineering and Computing > School of Computing
Research Initiatives and Centres > INSIGHT Centre for Data Analytics
Published in: 2019 International Conference on Content-Based Multimedia Indexing (CBMI). . IEEE. ISBN 978-1-7281-4673-7
Publisher:IEEE
Official URL:http://dx.doi.org/10.1109/CBMI.2019.8877392
Copyright Information:© 2018 IEEE
Use License:This item is licensed under a Creative Commons Attribution-NonCommercial-Share Alike 3.0 License. View License
Funders:Irish Research Council (IRC) under Grant Number GOIPG/2016/741
ID Code:24674
Deposited On:23 Jun 2020 12:37 by Cathal Gurrin . Last Modified 15 Dec 2021 15:46
Documents

Full text available as:

[thumbnail of CBMI2019_Liting.pdf]
Preview
PDF - Requires a PDF viewer such as GSview, Xpdf or Adobe Acrobat Reader
45MB
Downloads

Downloads

Downloads per month over past year

Archive Staff Only: edit this record