You’re About to See This Trippy AI Video Filter Everywhere—What Is It?

This is about much more than Facebook.

|
Oct 28 2016, 9:00am

Screengrab: '2001: A Picasso Odyssey' by Bhautik Joshi

You're about to see a lot of videos of family dogs rendered in Picasso or Van Gogh's signature painting styles on your Facebook feed, if Mark Zuckerberg's Facebook post on Monday apparently teasing a new photo app is any indication.

"I took this impressionist video of Beast [Zuck's dog] on my phone with a new AI technique called 'style transfer,'" Zuckerberg's post reads. "The idea is you show the artificial intelligence a painting and then it draws your photos or videos in that style in real time. Looking forward to getting this in your hands soon!"

So, it looks like Facebook has a new AI gadget to create shareable content—but the technology behind it, style transfer, is no toy. It's much bigger than Facebook or even Google, which also posted its own blog on style transfer research on Tuesday. In fact, style transfer is a rapidly accelerating field that stands to revolutionize visual design in everything from fashion to video games.

Read More: How MIT Is Teaching AI to Scare Us

The basic idea behind style transfer—taking the defining features of one thing, like the brushstrokes of a painting, and having a computer translate it onto another thing, like a photo—has been around for a while. But last year the technique got a major shot in the arm from a team of researchers from the University of Tübingen. In a 2015 paper, they describe how self-learning software called deep learning can make the process faster and more powerful. Fast forward a few months, and apps like Prisma have refined the technique so that it works as photo filter.

But style transfer isn't just a novelty, it's also a tool. Video game AI developer and Creative.ai co-founder Alex Champandard has been working on open source style transfer tools for designers. In the near future, he told me in an interview, gaming studios could be using AI to render game worlds in a fraction of the time it takes now. And, it follows, at a fraction of the cost.

"You can take the first part of a world—you can texture, say, a two metre by two metre world—and use these techniques to apply it to the whole level," said Champandard. "Or you can take an existing level, perhaps from a previous game or expansion pack, and then apply a weathered style so that you come back to that same level and this time it's flooded and everything's been destroyed."

"The content creation bar is dropping, and the cost of creativity is dropping, so that means we can take these techniques and do the same thing we did with 10 junior [employees] much faster," he continued.

Any game developers on the bottom rungs of their studio reading that probably felt a slight chill creep up their spine, but the fact remains that style transfer is poised to become a workable tool to create visual content faster than ever before. Style transfer isn't just limited to images, Champandard added—you could imagine an app one day using AI to transform your voice into George Clooney's, he said.

Facebook's apparent use of the technology, and apps and services such as Prisma, are a novelty, said Leon Gatys, one of the University of Tübingen researchers who wrote the 2015 paper that piqued Google and Facebook's interest in style transfer in the first place. But it's exciting to see AI become good enough to be a consumer product, he added—that means it works.

According to Gatys, the most exciting thing about style transfer is its links to neurology and human thought processes. That a machine can reliably produce images that are interesting to the human eye demonstrates that at some level software is beginning to understand, or perceive, images similarly to humans.

"Using the best machine learning algorithms right now, they see the world partly—and I don't want to overstate this—in a similar way to humans, and this is why we can use them to transform images in a way that's exciting to humans," Gatys said. "Style transfer is just a demonstration of that broader scheme."

There's still a lot of work to be done on style transfer that simply can't be tackled by corporate entities looking to push a fun app to market, he said. For example, the AI that's used by Prisma—and likely Facebook, eventually—is a one-shot deal. You put an image in, and the network calculates everything it needs to all in one go. It's fast, but it doesn't result in the highest-quality style transfer. The technique pioneered by Gatys and his crew, however, takes a slower iterative approach that produces images of much higher quality than what an app can pump out.

And that's part of the reason why apps like Prisma, and whatever prototype Zuck used to create his dog video, mostly render things in impressionistic painting styles.

"Impressionist paintings or anything with brush strokes that aren't photo-quality are easier because you can get away with a lot," Champagnard said. "You can insert brush strokes here and there that cover up deficiencies in the technology in a way."

When style transfer gets into Facebook's hands, we're likely to have a fun tool that will use a neat trick to turn regular photos or videos into fake masterworks. But for style transfer, it's only the beginning.

Get six of our favorite Motherboard stories every day by signing up for our newsletter.

Stories