A Brief History of Artists Turning Pictures into Music and Vice Versa

A week ago, YouTube star Andrew Huang turned an image of a unicorn into sound. His "Glorious MIDI Unicorn" illustration became a sort of synthesized piece of 17th century Baroque music. As Huang explains in his viral video tutorial, his MIDI artwork was inspired by musician and artist Aleksander Vinter, whose Baroque and Bach-influenced slice of music resembled a bird and sounded incredible. But turning images into sound and even editing photos with music software is nothing new.

Looking back, oscilloscopes have long been used to turn musical wavelengths into hypnotizing graphical moving images. But a proper starting point for discussing modern computerized image-to-sound file transfer might just be the work of Iannis Xenakis, a highly influential and original Greek composer who passed away in 2001. Back in 1977, Xenakis created UPIC, a digitizing tablet that connected to a computer, allowing him to effectively "draw" music. And it predates MIDI by at least six years.

Aphex Twin (Richard D. James), who once made a strange reference to UPIC, famously hid a rather satanic-looking version of his face in his song "[Equation]". To do this, he ran this demented selfie through the spectrogram software MetaSynth, turning the photo into MIDI notes that were very much in his musical style. Other pieces of standalone software like Coagula, RGB MusicLab and HighC (billed as a modern UPIC) can perform these audiovisual tasks as well.

Venetian Snares also explored image-to-sound file conversion in "Look," a song that seems to contain an image of cats. Plaid's experimental track "3 Recurring" contains a spectrogram of what look like a series of interconnected threes repeating from left to right.

Maria Mannone, a PhD candidate in music at the University of Minnesota, has also been transforming images into musical compositions. She associates each dimension with a musical parameter, so that length is time, width is volume, and height is pitch, amongst other arrangements. Like Vinter and Huang's MIDI artworks, Mannone makes a limited selection of points using Mathematica software to "make better sounds, easier music to play, while keeping the overall shape of the object."

"Once I write the score, I can convert it into a MIDI file via a notational software," says Mannone, who notes the process can also be used to turn sound into images. "We can finally visualize the content of the MIDI and see the image (only 2D, though). For the score of 'Eiffel Tower,' I selected 900 pixel-coordinates of one side of the building, transferred them into Mathematica software, created the MIDI file, and finally exported it into Sibelius and listened to it."

Another example is the artworks made by the Black MIDI subculture. Originating back in 2009, Black MIDI artists use software to create musical compositions stuffed with a seemingly impossible number of notes. They also design visuals, with colors connected to specific notes that created complex, psychedelic patterns when played.

Vinter and Huang's MIDI compositions share some DNA with Black MIDI art, but they aren't about speed or patterns. In a way, Vinter's bird and Huang's unicorn are more like a blend of spectrogram compositions and Black MIDI.

"It's a test of technical skill and imagination, combining musical knowledge about chords, arpeggios and octaves with visuals," Vinter tells Creators. "My MIDIs are based on Bach, using counterpoint, mirroring and playing with reflecting and opposing symmetrical shapes or patterns."

"Just like low-fi and pixel art, it has low resolution, so both creator and audience must use their imagination to recognize the image," he adds. "Some sequencers allow the use of colors within the 'piano roll' and more can be accomplished."

Vinter says that MIDI art isn't a movement or even a recognized art form. And for him, turning images into sound can often be more about how the composition looks rather than how it sounds. Vinter has only made a few himself, and most of them in a single day, including a monkey face, a Darth Vader face, a T-Rex, a fire-spitting dragon that plays the Mario theme, and a bird in the rain.

"The problem of going for the 'catch of the eye' approach with something like this is that you will encounter things like it sounding crap most of the time," Vinter says. "If you start with how it sounds and work from pattern recognition, not pattern creation, you got the idea of something that both sounds beautiful and looks cool. Of course, any way you want to approach it is fine as long as it looks and sounds good."

Related:

The Internet Is Freaking Out Over This Song Made from a Unicorn Drawing

Here's What Happens When You Edit Photos Like Music

Custom Instrument Turns Black-and-White Drawings into Sound