Watch These AI Hands Attempt to Play an Unplayable Song on the Piano

Learning to play an instrument is all about gaining muscle memory and teaching your hands to move in a way you didn't previously think they could. I took piano lessons for years, and eventually abandoned it as stubborn teens do, but I’d like to imagine that if put in the time eventually my hands would be able to move like the hands in this video:

But the hands in this video aren't real. They're a 3D rendering that’s programmed to move across the keys of a piano accurately to the tune of any song—except for ones that aren’t humanly possible to play. 

Created by Canadian-based Massive Technologies, the AI pianist is trained to listen to musical compositions and recreate them with virtual hands—and the results are pretty good. 

Fayez Salka, co-founder of Massive Technologies, told me that their initial goal was to develop an “AI-powered virtual piano teacher” that could help students learn to play. 

 The hand’s knuckles are a little too smooth and move with a bit less fluidity compared to the real thing, but it’s close, and weirdly mesmerizing. In one demonstration video, the AI pianist plays tunes based on audio input from clips including the Tom & Jerry “Cat Concerto” episode ("Hungarian Rhapsody No. 2"), as well as music from Pixar’s Soul. 

It also tries to play a “black midi” piece—compositions that can include billions or trillions of notes that are supposedly unplayable by two human hands—but ends up twitching across the keys in a flurry of noise and glitches while trying to keep up. 

To generate the AI pianist’s movements, Massive Technologies gave the algorithm an audio file as an input, and it outputs an animation file “complete with the correct playing technique and kinematically accurate hand and body animation,” Salka said. The engineers can then apply that file to a virtual avatar in a virtual environment—videos like what’s on their YouTube channel, augmented reality that lets the user place a pianist at a real-life piano, or virtual reality. 

To train the AI, Salka said his team brought professionally trained pianists to their labs in Helsinki, where they were asked to simply play the piano for hours. The AI observed their playing “through special hardware and sensors,” he said, and throughout the process the pianist and machine learning engineers would check the AI’s work and give it feedback or corrections.

“We would then take that feedback and use it as the curriculum for the AI for our next session with the pianist,” he said. “We repeated that process until the AI results closely matched the human playing technique and style.” 

Massive Technologies launched an AR pianist app in 2017 that puts a virtual piano player in the room with you, but that app only offered a selection of pre-set songs. The company is releasing a new app that incorporates the AI pianist’s listening skills—it’s in the Apple Store now, but still in beta. Salka said that soon users will be able to point their phone’s microphone at any audio source, and the app will show a virtual piano player performing that song. He said he also sees potential for piano teachers to create interactive virtual lessons for remote teaching—or for virtual piano concerts, and film or games creators who want to incorporate a super-realistic pianist in their scenes.