This story is over 5 years old.


Cosmic Visuals React to Live Audio in VR Performance 'SONO'

A performance by band LIVYATANIM gets transformed into a binaural spatial audio experience.

GIF courtesy of Or Fleisher and LIVYATANIM

Virtual reality and music make for a fruitful combination, and a new spatial audio pieceby Or Fleisher and Aviv Meshulam sees Meshulam's band LIVYATANIM performing live using binaural sound to add an extra level of immersion to the VR experience. The pair have previously collaborated on Mythan audiovisual VR film that let users explore music as topographical world populated by glitching whales and birds.


For this new venture, SONO, Fleisher and Meshulam wanted to captured a live performance in a unique way one that not only immersed the viewer but also featured a number of audio-reactive visuals. As LIVYATANIM perform, they appear to be floating in space on a small rock ("We chose the moon as our venue," notes Meshulam) while celestial happenings, reacting to the music's rhythm, take place around them.

"We decided to use lights that react to each musician’s musical role," notes Fleisher. "Emphasizing it and visualising the spatial audio element in the experience. We used the musical parts to drive the environment in the form of non-realistic massive celestial events."

Copernicus Crater by juniorxsound on Sketchfab

The piece can be experienced online in a browser with or without a VR headset, although it's recommended that you wear one if you've got it. To create the virtual moon landscape setting, the creators called upon NASA’s open source library of 3D scans, focusing in particular on the Copernicus Crater (above). The musicians' performances were shot separately using a cinematic stereo shooting rig and processed into their 3D environments, while the audio-reactive elements were built in TouchDesigner.

"After exploring how music and VR can merge into a cohesive and natural piece in Myth, binaural sound was an obvious step forward for making VR experiences truly realistic and immersive," explains Fleisher. "So naturally we wanted it to be an integral part of SONO. The project began as a desire to capture a live performance of the band in a new way, one that puts the audience in the center of things both visually and sonically."


Learn more in the making-of video below.

You can experience SONO online here. Download LIVYATANIM's new album here, and learn more about Phenomena Lab's work at their website.


Playable Music Video Is Like 'Fear and Loathing' in Virtual Reality

This Guy Just Spent 48 Hours in Virtual Reality

How Virtual Reality Is Changing the Way the UN Thinks