Sound design plays a pivotal role in VR games in helping immerse players within the experience. Alongside playing important gameplay roles such as providing audio feedback cues for interactions and guiding users through the virtual space. The all-encompassing nature of VR poses some unique challenges for an audio designer, in addition to all the familiar challenges of designing and implementing audio for games.
Now that I’m working on my second VR title as a sound designer, I have gained some insights into these challenges and some possible solutions for them. This is not a definitive list by any means, but the challenges listed here have proved to be common across the VR games I’ve worked on. The solutions are also not prescriptive, just some approaches that have worked for me.
Challenge 1: Spatial Audio and 3D Sound Positioning
Spatial audio is a crucial component in creating an immersive environment for VR games. Get the spatial audio right and players will feel completely immersed in the VR space. But get it wrong, and it could take players out of the experience altogether. Or, even worse, could make the user feel uncomfortable and want to take off their headset. VR games have the unique consideration of the end user's comfortability. More of which I’ll go into further detail later on in the article.
Because VR games completely encompass the player with sight and sound the audio has to be spatially accurate. Players can walk right up to sound emitters and interact with them in a 3D space, while feeling like they are actually there. Therefore they will instinctively know if something feels off with the audio, even if they might not be able to articulate what it might be.
So setting the attenuations, reverbs and occlusions in a VR game have to be done with great care and accuracy. It’s difficult to overstate the importance of this, especially if you haven’t experienced VR firsthand before.
Solution: Audio Middleware and using their Advanced Algorithms
One of the most powerful solutions to these spatial audio challenges is by using audio middleware, such as Wwise or Fmod. Their advanced spatial audio capabilities and algorithms can make it much easier to deploy convincing spatial audio than using a game engine’s default audio system.
As one example, Obstruction and Occlusion are terms used when a sound emitter is blocked or partially blocked by a game object. Such as when a wall is between a sound source and the audio listener.
Obstruction and Occlusion can be quite complex to set up and involve many complex calculations. But by using audio middleware like Wwise, and utilizing its rooms and portals system, it can make it much more straightforward to implement. It will handle a lot of the complex calculations in real time, without the need for a lot of programming resources to get it set up and running.
In addition, the reverb options available within Middleware can be more flexible and powerful than the reverbs that come with game engines. Allowing the sound designer to fine tune the acoustic profile of spaces in VR much easier and with greater precision.
Trying to replicate all of the spatial audio features available in audio middleware inside a game engine's stock audio system would likely be very time consuming and would require a significant amount of engineering and programming resources to set up.
Challenge 2: User Comfort and Avoiding Motion Sickness
VR games have the somewhat unique challenge of having to be aware of the end user's comfortability and susceptibility to motion sickness while playing. While it is doubtful that audio will directly contribute to motion sickness, inconsistent audio cues and mismatched audio-visual stimuli will have much more of a negative effect in VR than they would in a non-VR game.
So it is essential to consider user comfort while designing audio for VR games. Even among experienced VR users, there are still a large number of people who can only play VR games for short stretches of time at once. Therefore, being mindful of this susceptibility to discomfort during our audio design process will ensure that we are not contributing to any feelings of discomfort or unease for players while they are playing our game.
Maintaining consistent audio cues that match interactions and visual cues are very important in VR titles. VR games typically offer lots of physics interactions that the player can engage in, above and beyond what would be typically found in most flat-screen games.
From touching and picking up objects with their hands, swinging weapons that are velocity-mapped to their physical swing, to the handling of firearms that have real real-world based loading mechanisms. Which means that you often have to manually reload your weapon in exactly the same way you would their real-life counterparts - whereas in non-VR games you typically reload your weapon with the click of a button.
Solution: Consistent Audio Cues and User Testing
One solution to consistent audio cues might mean recording more of your own original sounds. I have often found that commercial sound libraries do not always contain the sounds I need when working on VR games. The added interactive element of VR games often means that I can’t find the performances that I’m looking for in sound libraries. Either because not enough variations are provided, or a particular object and surface combination has not been included or because an action is typically not available in non-VR games - so there hasn’t been a demand to provide these recordings in sound libraries. One example of this is dragging melee weapons across different surfaces in VR games. Often, melee weapon sound libraries do not provide these types of recordings and if they do, they might only be for one surface type.
Another solution is by being mindful that we do not create any harsh or fatiguing sounds in our audio design as it is doubly important in VR that no audio contributes to a feeling of discomfort or uneasiness with players because of the hyper sensitive nature of VR as a platform.
The most powerful way we can ensure that our audio cues match interactions and that we are not contributing to any feelings of discomfort is by extensive user testing. User testing is a valuable and necessary part of all game development. As sound designers, we are likely not going to be testing in the headset as often as we’d like. Time spent recording, in our DAW and then perhaps in middleware doing implementation, means that we often have limited time testing in the headset. So a sound that might sound totally harmless after some preliminary testing by us, might become fatiguing or uncomfortable to listen to after many hours of play time. Something that only extensive user testing will bring to light. Which we then can act upon.
Challenge 3: Balancing Sound Quality and Performance
A large majority of the VR gaming market are playing on standalone headsets, like the Meta Quest. Standalone headsets, while very impressive in what they can achieve, are not as powerful as current generation game consoles or anywhere near high end PCs in their power and performance. Therefore, there is often going to be a delicate balancing act between quality and performance.
For audio this means that resources will have to be managed carefully. Which will have bearings on how we compress the game’s audio assets and the quality and amount of real time audio processes we can use. That will impact things like the quality of in-game reverbs, for example.
Solution: Efficient Audio Asset Management
Efficiently managing your audio assets and sound systems is key. Like choosing audio compression settings that find a balance between sound quality and size footprint for your assets.
Another solution might also mean reusing assets wherever you can. If you have some common actions in the game, is it possible that they can share the same set of sounds? Knowing this beforehand will help, as it will influence the design phase of the sounds if you know that they have to be suitable for multiple sets of game actions.
Baking-in reverb to some assets might be a more efficient way to handle reverb in some specific situations. This is frequently done on gunshot tails, for example, and a system will switch out the tail to reflect the environment the gun was fired in - like a small or large room or outdoor space.
Baking-in reverb can often mean higher quality reverb, as the quality and amount of reverb plugins available in our DAW’s are more numerous and powerful than those ordinarily available in game engines or audio middleware.
But this can’t be done for everything. At some point you will have to rely on in-game reverb. Then you will have to walk a fine line between the quality of the sound of that reverb versus the system resources that it uses. One good approach to this is being mindful of areas where you can get away with lower quality reverbs (perhaps in a space that the player only passes through briefly) so that you have the bandwidth for higher quality reverbs in more important areas of the game.
Conclusion: Sound Design in VR and AR Games
I hope I’ve shown that sound design plays an important role in VR games in fully immersing players in the virtual space. But it also comes with a unique set of challenges that sound designers for VR games must contend with: accurate spatial audio, a marriage of audio and visual stimuli, minimizing player discomfort and being aware of the technical challenges and limitations of VR hardware are all important things to consider.
But with attentive consideration of these challenges and employing solutions such as audio middleware and extensive user testing, your audio will greatly contribute to making your VR game feel immersive and great to play.
Do you need an experienced sound designer for your VR game? Get in touch and tell me about your project today!
The Author
This article was written by Oliver Smith. A long-time remote working freelance sound designer dedicated to making gameplay enhancing sound for games.