Pollen Music Group has done sound design and composed the music for over two dozen virtual reality films now, including Glen Keane’s Duet and Patrick Osborne’s Pearl, which is the first VR film to be nominated for an Academy Award. Most recently, it created Sonaria, a VR experience for the Google Spotlight Stories platform. Partnering with Chromosphere, it’s the first time Pollen has both directed the visuals as well as the sound. It’s available now on iOS and Android through the Google Spotlight Stories app, as well as on HTC Vive and the Oculus Rift through Steam.
Sonaria follows the journey of two seals in a colorful world filled with geometric creatures. It has a pleasingly minimalist aesthetic, and it’s meant to be a relaxing experience that’s largely driven by the environment’s sounds.
Pollen started in 2010, originally composing for traditional TV and film such as Pixar’s short Presto. In 2012, Presto‘s director Doug Sweetland was working on some projects for Google’s Advanced Technology and Projects (ATAP) group, and he introduced Pollen’s principal/composer Scot Stafford to the idea of 360-degree storytelling.
“I’d done some work in games and had an idea of what they might need, but I went in and met with [Sweetland] and Karen Dufilho, the executive producer at what is now Google Spotlight Stories,” said Stafford in a phone call with GamesBeat. “They showed me some tests where you used a phone like a magic window to a fully spherical 360 world all around you. We were following a mouse and a hat as it was involved this chase, as the wind is blowing the hat away from the mouse. I’d never seen anything like it.”
Since then, Pollen has worked on a number of VR projects for Spotlight Stories, which is an iOS and Android app that features VR films that viewers can experience using headsets like the Google Cardboard or Daydream. Sonaria came about because Pollen wanted to explore how sound and music can be even more immersive with six degrees of freedom in VR. Stafford wanted to put users on a “threshold between two worlds,” which they could traverse by just standing or sitting. It’s something that Pollen has experimented with before, such as in Pearl when the viewer is inside a car and can stand up with their head through the sunroof.
“So Sonaria started with, ‘What’s another type of threshold we could be on?'” said Stafford. “I came up with this idea of being underwater below the Antarctic ice shelf and following seals as they swim around below us, and then surface onto the top of the ice shelf. We can go up there with them. That turned into this journey as we follow these two creatures as they transform from simple life forms to more and more complex ones.”
To create the “stained-glass” aesthetic, Pollen partnered with Chromosphere and Kevin Dart, who was the creative director on the short film June. Stafford composed a brief piece of music to give Dart an idea of the kind of mood he was going for in Sonaria, some audio recordings of seals, and a quick sketch of a creature made up of teardrop shapes.
“The idea was to create a really minimal visual design language, and Kevin latched onto the teardrop shape and said, ‘What if this was the building block of life in this world we’re creating?'” said Stafford. “And so we went back and forth. He would send me these amazing designs of creatures, all over the map. How few teardrops do you need to make people think you’re looking at a frog, at a jellyfish, at a whale? It was this neat exploration of how much we can do with very little, visually, with the idea that the sound and music would be a lot denser and richer and fill in all the lines.”
Creating the tools
Spherical sound design for VR has been a unique challenge, especially since Pollen had to create some of the tools it needed. Pollen lead recording engineer J.J. Wiesler says that at the start, it found a few pieces of software it could use. However, those were mainly intended for academic purposes and didn’t fit into the kind of workflow that sound designers and composers require. Spotlight Stories hired Pollen to develop its own tools, audio plug-ins that could be compatible with the in-house editing tool as well as industry standards such as Avid Pro Tools. It did so in partnership with Paul Neyrinck, whose company Neyrinck specialized in audio software.
“We started using them ourselves. It made things a lot easier,” said Wiesler. “We continue to evolve them, because as Scot mentioned, three degrees of freedom is now moving to six degrees of freedom. Because it’s so important that, when a user walks around an object that’s making sound, that sound stays attached to that character or object. And there still, to this day, is really no way to work with objects in a typical composer or sound designer’s environment.”
In VR, it’s crucial that the sounds create not only an immersive environment but convey enough information about where the viewer should look. Stafford says that half the challenge of VR sound design is technical and half is creative. On the creative side, it’s the fact that you’re designing an entire world but also culling it down so that it’s not sensory overload.
Like in traditional sound design, Wiesler says that Pollen has had to decide between realism and what it wants the viewers to hear, though VR is trickier because the sphere of sound is constantly going. But along with the challenges comes an added opportunity to use sound to direct the viewer’s attention and move things along.
Space and time in VR
“I think that VR asks us to at least keep in mind the physical and spatial reality of things a lot more than film does,” said Stafford. “It’s something you have to be really sensitive to. As far as how to choose which layers to focus on, choosing what to sound design, there’s also the way we mix things in real time to your headphones. There are ways to focus a mix. We can choose, for every sound, do we want it to taper off and become quieter the further off screen it goes? Is it such an important story point that we need it to be at full volume no matter what?”
One of the key points in Sonaria is crossing the threshold of dry land and underwater. The sound is completely different underwater, and Stafford says the team had to create new techniques for creating the transition when a viewer submerges or re-emerges from the ocean.
“I think we had seven different sounds for submerging and surfacing that would be triggered based on how quickly you stand and sit,” said Stafford. “We also needed to create what’s called procedural reverb, meaning it’s happening in real time. These are all real time effects. Nothing can be baked in. We needed to have the sound of things being very dry above the water, and then very wet and reverb-y below.”
Transitions contribute to the complexity of composing music for VR. It’s one of many challenges, which include trying to create a seamless experience, timing the emotional buildup when the player’s behavior is unpredictable, and choosing between constant music versus moments of silence to allow background sounds to emerge.
“On the one hand, it’s nothing new. Mozart wrote nonlinear compositions that relied on the roll of dice, back in the 18th century,” said Stafford. “More recently we’ve had video games with time elasticity. You could easily say, with good reason, that video games have achieved a really theatrical experience. Many composers have tackled this challenge and done some amazing work. I would say it’s slightly different when you get into a pure film experience. You’re not doing anything. You’re a witness. This is what’s called gaze-based interactivity, where all we care about is where you’re looking.”
What’s next
“We’re working on several projects right now. We can’t talk too much about them, but I would say that the move from three degrees of freedom to six degrees of freedom is a challenge that we’re still having a lot of fun trying to come up with different answers every time,” said Stafford.
The team is also looking forward to experimenting more with augmented reality and mixed reality. These offer a whole new way of looking at sound design, since it incorporates what’s happening around the player in the real world.
“I think there’s a long way to go for that, but it has a ton of potential,” said Wiesler. “Because it’s less isolating. The fact that you can hear all the sounds around you helps it be less, I don’t know—it makes you less uncomfortable, while VR still has that challenge to overcome. You can share the experience with a friend.”
No matter what challenges Pollen tackles next, it will continue making its own tools along with creating new sound-focused experiences.
“As a creative working in this world that’s also creating tech, it’s a bit like being the engineer of a train, and also throwing the tracks down in front of you as you go,” said Stafford.”