Login (DCU Staff Only)
Login (DCU Staff Only)

DORAS | DCU Research Repository

Explore open access research and scholarly works from DCU

Advanced Search

A CNN-based Framework for Enhancing 360° VR Experiences with Multisensorial Effects

Szabo, Peter, Simiscuka, Anderson Augusto orcid logoORCID: 0000-0002-0851-2452, Masneri, Stefano orcid logoORCID: 0000-0003-1965-5704, Zorrilla, Mikel orcid logoORCID: 0000-0003-2589-2490 and Muntean, Gabriel-Miro orcid logoORCID: 0000-0002-9332-4770 (2022) A CNN-based Framework for Enhancing 360° VR Experiences with Multisensorial Effects. IEEE Transactions on Multimedia . ISSN 1941-0077

Abstract
Improving user experience during the delivery of immersive content is crucial for its success for both the content creators and audience. Creators can express themselves better with multisensory stimulation, while the audience can experience a higher level of involvement. The rapid development of mulsemedia devices provides better access for stimuli such as olfaction and haptics. Nevertheless, due to the required manual annotation process of adding mulsemedia effects, the amount of content available with sensorial effects is still limited. This work introduces an innovative mulsemedia-enhancement solution capable of automatically generating olfactory and haptic content based on 360° video content, with the use of neural networks. Two parallel neural networks are responsible for automatically adding scents to 360° videos: a scene detection network (responsible for static, global content) and an action detection network (responsible for dynamic, local content). A 360° video dataset with scent labels is also created and used for evaluating the robustness of the proposed solution. The solution achieves a 69.19% olfactory accuracy and 72.26% haptics accuracy during evaluation using two different datasets.
Metadata
Item Type:Article (Published)
Refereed:Yes
Uncontrolled Keywords:Multisensory media, neural networks, image recognition, olfaction, haptics, machine learning
Subjects:Computer Science > Image processing
DCU Faculties and Centres:DCU Faculties and Schools > Faculty of Engineering and Computing > School of Electronic Engineering
Research Initiatives and Centres > INSIGHT Centre for Data Analytics
Publisher:Institute of Electrical and Electronics Engineers
Official URL:http://dx.doi.org/10.1109/TMM.2022.3157556
Copyright Information:© 2022 The Authors.
Funders:EU Horizon 2020 Research and Innovation programme under Grant Agreement no. 870610 for the TRACTION project, Science Foundation Ireland (SFI) Research Centres Programme Grant Numbers SFI/12/RC/2289_P2 (Insight) and SFI/16/SP/3804 (ENABLE)
ID Code:26879
Deposited On:01 Apr 2022 11:13 by Anderson Augusto Simiscuka . Last Modified 24 Mar 2023 14:25
Documents

Full text available as:

[thumbnail of A_CNN-based_Framework_for_Enhancing_360_VR_Experiences_with_Multisensorial_Effects.pdf]
Preview
PDF - Requires a PDF viewer such as GSview, Xpdf or Adobe Acrobat Reader
Creative Commons: Attribution 4.0
17MB
Downloads

Downloads

Downloads per month over past year

Archive Staff Only: edit this record