Why Are We Still Listening to Music in Two Dimensions?

In the last couple years, Paul McCartney, Beck, Coldplay, Jack White, the LA Philharmonic, and many other musicians have put on virtual concerts, recreating a live performance with not only 360-degree video, but three-dimensional sound that captures an acoustic scene exactly as we'd hear it in real life.

Audio is crucial to convincing the mind it's in a different reality, because sound is essential to how the brain orients in space. Without 3D audio that faithfully mimics reality, the virtual illusion is lost. But immersive lifelike audio can transport you to a scene, to make you feel like right there in the studio with a band or in the front row of a huge concert hall.

The same way virtual reality video takes you out of the confines of the square screen, 3D audio could blow up the conventional paradigm of music coming from a stage in front of you. Why settle for a realistic simulated experience of a live orchestra when you can be listening from the middle of the strings section instead, violins circling around your head, or from the perspective of the performer up on stage?

Put another way, are we going post-show?

"In 10 years you could be living in Tahiti as a surfer and you'll go to your living room and say, 'Tonight should I see the Philharmonic? No? The Paris ballet? The Stones in Madison Square Garden?' And you're there," David Chesky, founder of the New York-based audiophile label Chesky Records, told me.

"But I'm sure some young director's gonna say, 'you know, this is like really stupid, we could do hipper things. Like, why don't we do the concert in like, the Gobi desert in surround under the stars?'" Chesky added. "Imagine if you could be sitting on an island in the middle of the ocean with a symphony around you and watching the whales around you and the stars, hearing a concert, and you're there."

"People could take it anywhere. Whatever the imagination can think of."

While 3D audio is an obvious boon for virtual concerts, it can also change how we listen to recorded music. For decades, musicians, including luminaries like Lou Reed, Pearl Jam, and Radiohead, have dabbled with adding a third dimension to their recordings. But 3D sound never quite went mainstream. There just wasn't a huge appetite for audio that sounded like real life.

Now, there is. Three-dimensional audio is a hot topic again ever since VR industry-types wised up to the fact that immersive video is never going to be believable without its sonic better half. As the VR market scrambles to perfect the technology to immerse listeners in an acoustic space, it will no doubt pull music along with it.

"We're going to start seeing this sort of new genre of audio experiences," sound installation artist Gabe Liberti told me in a recent phone conversation. His interactive music installation appearing at the upcoming South By Southwest festival, where a handful of panels will discuss 3D sound this year. "Putting people in spaces hasn't really been part of the paradigm of producing music. So that's completely new."

Three-dimensional "spatial" sound adds the dimension of depth to recorded audio, instead of just panning horizontally between left and right ears like conventional stereo tracks. This puts sounds in space: Sounds seem to be coming from not just above, below, and all around you, but also right up close or further in the distance, in every direction, as if you're enveloped in an acoustic sphere.

Humans are naturally able to hear in 3D, even though we only have two ears, because of the way the ears are positioned on the head. A sound coming from the right will hit the left ear a fraction of a second later than the right ear and will be perceived slightly softer as the sound diminishes and is reflected off the outer ear and head and torso. These aural cues allow the brain to identify the direction of a sound and pinpoint its spatial location remarkably well—it's what makes you whip your head around when you hear a twig snap behind you.

The most well-known technique for capturing 3D audio, binaural recording, mimics this natural process by putting microphones in fabricated ear canals on either side of a dummy head. (Or with tiny mics you stick in your ears like these Hooke headphones.)

Binaural recording has been around a long time—it's how Lou Reed played around with spatial audio back in the day, and musicians are still experimenting with it. There are 3D binaural tracks on Reddit and Soundcloud and YouTube and Vimeo. If you're not familiar with the effect, this video is a good example. Chesky Records has a whole series of binaural albums recorded with a dummy head in the studio, and was nice enough to give Motherboard readers a free sample of some of its binaural music.

So why aren't we all listening to music in 3D?

Important to note is that binaural audio only works with headphones, not speakers. This has seriously limited its commercial potential in the past. Since the recording relies on the spatial positioning of the two ears, sounds have to reach only the intended ear. But with speakers, the sounds spread and get mixed up; the right ear hears the sounds meant for the left and vice versa—a problem called "crosstalk."

But give it a year or so and this could change. A professor at Princeton University's physics department, Dr. Edgar Choueiri, invented a way to make binaural audio work on speakers, which could mark a huge step forward for 3D music.

The technology, called BACCH 3D sound, solves the crosstalk problem by putting a real-time digital filter between the two ears to mimic the natural wall a human head or dummy head provides. (Here's a great video of Choueri explaining how it works.) The filter device currently costs around $50,000 (and a few have sold), but the goal is to make an affordable consumer version. Princeton has already licensed part of its 3D sound technology to Jawbone to power the Jambox speaker's "LiveAudio" feature.

Chesky, who has partnered with Choueiri, predicted a consumer version of the filter would be ready in about a year to let listeners experience 3D binaural sound on speakers.

"And man, it's much better than headphones," he said. "It'll blow you away."

For now, spatial audio is still confined to niche and experimental musical circles. I spent the last few days listening to binaural recordings (on headphones) and while they do sound more acoustically rich and textured than conventional stereo, and there were some eerily realistic moments, they weren't exactly transportive. Though I will say sounds that seem to be coming from the back of the head are very chilling. They make my hair stand on end every time. So it's no surprise that binaural audio has found a natural early home in the ASMR community.

Now imagine the same effect, only it's Justin Bieber singing sexily right next to your ear. You can start to see the potential spatial music has for increasing intimacy between artists and fans.

Because binaural audio captures the music as it sounds in real life, it's also great for creating unique acoustic aesthetics, like the echoey reverb of the New York City Subway or a woodsy folk atmosphere.

"You're just in this space," said Chesky, describing a newly released binaural album by City of the Sun. "It has dimension, it has texture, it has density. So it makes the experience a better roller coaster ride."

However, traditional binaural audio has a few pitfalls that make it stop short of being true immersive 3D sound.

Pitfall No. 1: It's recorded for just one head position, so when a listener moves her head around, that acoustic scene comes with her, and that destroys the sonic illusion. The listener can't hear a different orientation inside the sound field the way she can move her gaze around a VR image.

Pitfall No. 2: Everyone's ears are shaped differently, like unique fingerprints, but a dummy head is just working with a general ear shape. The difference between the dummy ears and the listener's ear shape docks the acoustic accuracy of binaural audio.

The holy grail in truly immersive 3D sound is real-time customized spatial audio that is calibrated to the anatomical measurements of one's ears and uses head-tracking technology to update the soundscape as one moves their head around. "It really becomes real to you, and vivid, if it feels like you've been immersed in a new, living acoustic reality," said Liberti. "You feel like you're somewhere else."

Binaural "isn't really dramatic enough if when you move your head things don't change," he said. "I think this is one of the key parts to making 3D audio actually relevant."

The technology to create real-time immersive audio is being developed as we speak, dipping into other 3D techniques along with binaural, like ambisonics, holophonics, wavefield synthesis, virtual spatial audio, or object-based audio.

New Kickstarter-funded Ossic headphones claim to use head-tracking tech and calibration sensors in the headphones to measure the shape of a listener's ears and use that data to locate sounds for more accurate realism. And most of the major VR headset prototypes are also developing a format for immersive spatial audio. The emerging VR market will no doubt inform how 3D sound is incorporated into music.

Audiophiles who obsess over finding a $5,000 power cable to improve sound are "fighting for such small, small, small, arguably imperceptible changes in the fidelity," said Liberti, "but when you think about the difference between a stereo recording and a real-time, three-dimensional spacial recording or mix, the difference between those is so dramatic. And the opportunities for what we can do in mixing and moving sound around you and immersing you in sound is so vast… I can't even think about all the possibilities."

Over the years, music has increasingly put sound in space. First with the move from mono to stereo, and then surround sound, and then object-based audio like Dolby Atmos that can direct sounds to specific points in space. Spatial audio is the next sonic boundary to cross, and when we do we may look back on the days we listened to sound in two-dimensional stereo with disbelief, said Chesky.

"We're gonna look at it like you look at a 15-inch black and white TV with fuzzy lines watching Elvis Presley."