Acoustic Information Retrieval for Interactive Sound Rendering in Virtual Environments

Colombo, Mattia (2025) Acoustic Information Retrieval for Interactive Sound Rendering in Virtual Environments. Doctoral thesis, Birmingham City University.

[thumbnail of Mattia Colombo PhD Thesis_Final Version_Final Award April 2025.pdf]

Preview

Text
Mattia Colombo PhD Thesis_Final Version_Final Award April 2025.pdf - Accepted Version
Download (58MB)

Abstract

The adoption of Extended Reality across industry and research domains has incentivised the development of Head-Mounted Display (HMD) technology, driving the field towards better and more optimal techniques for efficient and realistic rendering, offering sensing and capturing capabilities. Virtual entities visualised through HMDs can interact with spatial features of the users’ surroundings, allowing for realistic context-aware interactions and improving task performance and the perceptual quality of the overall immersive experience. However, techniques for rendering realistic audio stimuli that respond to spatial features of the immersive environment are underrepresented, considering an extensive body of literature on Mixed Reality (MR) research domains. Perceptually valid sound rendering is key to realism in audio stimuli within immersive environments, as spatial features of the users’ surroundings can be considered, approximating fundamental characteristics of sound propagation in environments. This enables listeners to use natural hearing abilities that interpret sound propagation effects to sense space and entities in their proximities, affecting interactions in immersive experiences. This thesis reviews the current state of sound rendering techniques and their application and feasibility across several use cases, proposing, as a novel contribution, a pipeline that can generate context-aware realistic audio for MR applications. The development of this pipeline involves adopting computer vision techniques in the process of decomposing complex scenes to recognise acoustic characteristics of space, determining physical and structural features of the environment surrounding HMD users, and allowing audio stimuli to respond to spatial characteristics of the immersive environment. The experiments presented demonstrate applications of scene understanding techniques applied to virtual environments and reconstructions of real space to determine acoustic properties of surfaces and entities for automating the application of sound rendering. This is done by identifying the current state of automatic acoustic material recognition for virtual environments and proposing novel evaluation methods that test the efficacy of automatic systems for tagging acoustic materials in virtual environments. Proof-of-concept systems have been tested on state-of-the-art acoustic renderers to demonstrate their efficiency in real-world scenes. Participant testing using a prototype deployment of the proposed pipeline that measures the performance of psychoacoustics-related tasks suggests that audio stimuli generated using the proposed pipeline have a significant effect on task performance within Mixed Reality applications. Current directions are aimed at designing end-to-end pipelines for interactive, real-time applica-tions, with the ambition of adopting computer vision to understand the acoustic space, even in contexts of dynamic geometry typical of HMD technology, where the acoustic space is constantly updating based on the users’ surroundings.

Item Type:	Thesis (Doctoral)
Dates:	Date Event 15 April 2025 UNSPECIFIED
Uncontrolled Keywords:	Computer Vision for sound rendering, acoustic space reconstruction, psychoacoustic factors in sound rendering, sound rendering for mixed reality devices
Subjects:	CAH11 - computing > CAH11-01 - computing > CAH11-01-01 - computer science CAH11 - computing > CAH11-01 - computing > CAH11-01-04 - software engineering CAH11 - computing > CAH11-01 - computing > CAH11-01-08 - others in computing
Divisions:	Doctoral Research College > Doctoral Theses Collection Faculty of Computing, Engineering and the Built Environment > College of Computing
Depositing User:	Louise Muldowney
Date Deposited:	08 May 2025 11:41
Last Modified:	08 May 2025 11:41
URI:	https://www.open-access.bcu.ac.uk/id/eprint/16343