Holly Herndon Deepfaked Herself Into a 'Digital Twin' That Sings Any Song In Her Voice

Electronic musician Holly Herndon is a leading proponent of using artificial intelligence in art, creating an AI called Spawn—a “singing neural network”—with partner Mathew Dryhurst that was featured prominently on her 2019 album, Proto. Now, anyone can have Herndon sing back any audio recording with her voice… or rather, her “digital twin’s” voice.

Holly+ is Herndon’s new project, which is kicking off with an AI-driven vocal tool developed by Never Before Heard Sounds that was trained on recordings of her singing. It relies on machine learning to recreate the distinctive nuances of her voice. Anyone can upload a polyphonic song or voice recording to the Holly+ site, and within a few minutes they’ll have a version “sung” by her digital copy, much as if her voice was played like a synth.

Soon after launching, Herndon tweeted that there were 18.5 requests per second to her server, and that there had been 1,000 such song transformations within an hour. Want to hear the Neon Genesis Evangelion anime theme song tackled by an algorithm? Done and done. According to Herndon, more tools are coming and they’ll get more sophisticated in time.

The technology itself isn’t new: AI algorithms have produced compelling likenesses of voices for years, such as with a tool developed by Chinese tech giant Baidu in 2018. And there are tools that can create convincing deepfake likenesses of celebrity voices, like the one that rankled Jordan Peterson in 2019 and was quickly shut down after threats of legal action.

Part of what makes Holly+ unique is the initiative to turn her voice into a publicly-available instrument that anyone can use. The advent of machine learning tools that let anybody copy anybody else’s physical appearance or voice have led to widespread debate around things like bodily autonomy and intellectual property rights. Herndon is pushing back against the idea that allowing for a potentially wide array of derivative works with machine learning will devalue her own music.

In an extensive post about Holly+, Herndon explores the potential rights issues that could surface around deep learning voice models based on commercial artists. She cites cases of corporations tapping impersonators to copy popular songs for ads, such as Midler v. Ford Motor Co. in 1988, which concerned a series of commercials for the automaker that used an impersonator of Bette Midler. Herndon writes that “precedents suggest that public figures will retain exclusive rights to the exploitation of their vocal likeness for commercial purposes.”

Instead, she believes that as the digital likeness of her voice grows in use and popularity, it will only increase the value of her own original works. It’s the “My Collectible Ass” theory, she cites, which holds that “the more prominent and visible/audible a work of art is, the more valuable the certified original becomes.” Holly+ creations are made under an open-source license and can be freely released, but cannot be commercially exploited without permission.

She will not govern the commercial use of Holly+ creations herself, however. Instead, Herndon will create a decentralized autonomous organization (DAO), in which members are issued ERC-20 VOICE governance tokens on the Ethereum blockchain. Those token-holding stewards will vote on whether works created using Holly+ tools can be commercialized, and make decisions about licensing agreements. Herndon is no stranger to exploring the artistic potential of blockchains, including by creating NFTs of generative art pieces. 

Herndon told Motherboard that tokens will be distributed to friends, family members, supporters, and artists who contribute to the Holly+ platform, but that there will not be a public token sale. 

“I need to make sure people who are able to vote on official usage of Holly+ understand who I am and what I represent,” she said. Approved songs will be minted as a non-fungible token (NFT) and sold via a Zora auction house, with 50 percent of profits going to the new artist, 40 percent to the DAO members, and 10 percent to Herndon herself.

“Through this model I can give people tools to experiment with my voice, create an organization to vet and approve official usage, and generate funds to create new tools. That seems cooler to me than approaching it from a defensive DRM angle,” Herndon told Motherboard. “We need ways to compensate artists whose likeness is being used, certify what is ‘real’ and what is not, and also celebrate a new and strange capacity we have to perform as other people. I think the model we came up with covers those points!”

Zora, which was co-founded by former employees of cryptocurrency exchange Coinbase, enables creators to use DAO governance to develop brands and products in a new, collective manner. It’s part of a rising trend of DAO-led brand creation and management in the crypto space. Earlier this year, Motherboard spoke to Herndon about how blockchain tech has the potential to give power back to musicians. With Holly+, she seems elated about the creative possibilities ahead.

“I am excited at the potential for there to be many different Holly albums in the world. I have already received some questions from musicians to ask if they can make Holly+ records, and that sounds perfect <3,” she told Motherboard. “I am curious what it would feel like to be in a band with 10,000 people, and how that might interact with my personal music project.”

“I like the idea of licensing songs from Holly+ musicians to play in my own sets, or DAO members performing as Holly+ without me,” Herndon added. “The DAO will make all of that stuff possible. This is fundamentally an experiment, so the uncertainty of everything is pretty thrilling.”