FYI.

This story is over 5 years old.

Music

Dancing To The Sound Of Shadows

Sembler map the visual movements of shadow puppets, using visual data to generate a real-time soundtrack to accompany a performance.

If someone told you they were going to make music from shadow puppets, you’d think: How? Why? What sound does a shadow make anyway? But no, design group Sembler, in collaboration with shadow puppeteers Feral Theatre, used computer visioning techniques to transform movements from the Feral Theatre’s production of The Sound Catcher into a generative music piece. Taking the visual data and translating it into sound, providing an audio interpretation of the dynamics of the play.

Advertisement

Sembler explain their process:

Surveillance techniques for detecting, recognizing and tracking objects were re-appropriated to map the visual information of the performance into a musical domain. Using a custom built computer vision system that identified the movement of the puppets, the structure of the piece and the overall activity, we produced real-time, generative music that reflected Feral Theatre’s performance.

The piece took inspiration from gamelan, a kind of Indonesian music that has an association with shadow puppetry, and features a variety of instruments that are specifically built and tuned to be played together as a unit, creating a structured sound that relates well with algorithmic processes.

Also produced by Sembler, and worth mentioning, is the music video below for Darkstar’s track “Gold”. It’s interesting as it uses a variety of visual techniques, including particle engine animation, 3D point clouds, and MRI scan images sourced from The National Library of Medicine in Maryland, in subtle and intriguing ways. You can read more about how they did it here.