Face Hacking: Transforming Our Future Visages With Digital Makeup

Augmented reality could show one mask to your friends, another to your enemies.

Feb 24 2015, 3:00pm

​Lia Kantrowitz

In a studio on a quiet street in the Shibuya prefecture of Tokyo, Nobumichi Asai is hacking human faces. By combining design, motion tracking, and projection mapping, Asai is pushing the limits of human appearance with his "digital makeup."

"The face is very interesting media," Asai told me in Japanese as his wife confidently translated, as if she has had many discussions about this topic.

In October 2014, Asai, along with makeup artist Hiroto Kuwahara and French digital image engineer Paul Lacroix, released a video of Omo​te. Named so after the Japanese word for "face" or "external," the augmented reality project was first displayed in a live installation where different omotes were ​mapped onto real faces, which Asai once wrote he considers "the most delicat​e yet powerful medium for art."

In the demo, a woman sits in front an array of motion sensors, eyes closed, as her face is scanned from every direction. Her eyes then appear to open, but these eyes aren't seeing—they're a projection mapped over her closed eyelids. The model slowly tilts and moves her head while the projection moves along with her, the eyes, lips and contours of the makeup exactly where the viewer would expect them to be. A computer program seamlessly transforms the face through several makeup variations, many that cover the whole face, using artistic inspiration drawn from traditional Japanese masks.

Omote is both thrilling and a little distressing to watch, falling into that uncomfortable but not quite scary place in the uncanny valley. The internet was captivated by Omote; the video has over 6.3 million views to date, with comments ranging from "My jaw literally dropped" to " "How long before we're all walking the streets with AR masks?" Since the debut, Asai has made a simi​lar vid​eo featuring two Japanese television hosts.

Indeed, Omote demonstrates what the near-future of appearances could be in a more digital world where augmented reality (AR) is abundant. Where once plastic surgery was needed to change someone's looks drastically, we have just begun to tap into digital tools that could allow for much more radical image alterations faster, more easily, and fully reversible. Soon, having a skin for an augmented reality world could be as common as having a Facebook account is today.

In the same way users can define what different groups see on their online profiles, they could also define which augmented reality personas are seen by others: one for your prospective employers where you look like a respectable person ready to be gainfully employed; another just for your friends with full stage makeup straight out of a Lady Gaga video.

"The differences between fantasy and reality are becoming vague," Asai says.

Image: Asai via ​Design Bloom

Of course, real life avatars would undoubtedly come with the potential for abuse. Imagine someone hacking your AR avatar and tattooing a dick onto your forehead. Or creating an anonymity cloak that could obscure the wearer's identity "by making it impossible to describe or remember them," as Philip K. Dick described with his scramble suit in A Scanner Darkly. Digital anonymity is problematic enough in comment sections—what havoc could it wreak in real life?

We don't have to worry about it quite yet, as there are still technical problems that need to be addressed before this technology could begin to reach any kind of widespread adoption. For one, the motion tracking cannot keep up with fast movements, and current hardware isn't powerful enough to eliminate the latency that would occur in a system like Omote used in a real-time AR environment.

Time is another limiting resource. The first Omote demo took six months to develop with 10 people working on it. According to Asai's artist ​statement, they had to create a computer program in C++. For each new iteration, Asai and his associates must start fresh with a process that takes months. First, a scan must be done of the head of the model, which is 3D-printed. For the tracking technology, the head must then be painted the correct skin tone so it can be used as practice. Track markers are placed on the head and an artist goes into Photoshop and designs the images that will be projected, which are used to create CG renderings and meticulously matched to the projectors.

And of course, what good is anyone wearing an AR mask IRL if you have to keep your eyes closed to show it off? Reconfiguring Omote so that models' eyes can be open while the rest of the face is projected with lights would also be more work.

Asai said he hopes to speed Omote up and improve upon its technology this year. Meanwhile, others are working on similar projects as this segment of augmented reality is quickly changing. After all, the technology itself is not new or unique; researchers have been doing work that combines motion tracking and projection mapping for at least the last couple years.

British artist Jenny Lee has also used augmented reality to explore "the notion of being human." Lee created a project in 2012 that used augmented reality to project digital skins of her design onto people's faces. Calling the project The Future Human, Lee set out to think about what people could look like in 2060. What she came up with was not particularly humanoid.

"I didn't design it in a way to be aesthetically pleasing," Lee told me in an interview.

"It was shocking to people. They felt like their skin was changing."

For The Future Human, people stood in front of an interactive mirror and saw an overlay of the digital skins on themselves. Projections like jagged, rock-like structures, prisms or squiggly lines would seem to grow out of their faces.

"It was shocking to people," Lee says. "They felt like their skin was changing, it felt quite real to them."

But who is to say what the human aesthetic will be in 2060? Our current, part-time digital lives might be a closest thing to a sneak peek of how we will create a real life avatar. Lee points to how people currently use Facebook, where photos of vacations, status updates of good news, and other content is chosen and edited by the user to craft the digital identity they want—not necessarily a mirror of real life—and project it to others.

To date, AR technology has mostly been used for very practical purposes. Products like Google-backed Magic L​eap, Sony's recently announc​ed AR glasses, and the smart contact lenses in development are paving the way to a more digitally-enhanced life. Asai said these tools are part of the change that will make face hacking AR as common as the current variety of AR smartphone apps that can do things like overlay a map on your current location.

Outside of the smartphone industry, Microsoft Research's Illum​iroom project expands the playing radius of video games, expanding the screen around the room and immersing players in sights and sounds.

Whatever the future holds, AR will surely be a part of it. In what way exactly is up to the artists, technologists—and user demand.

This article is part of Bodies of the Future, a collaboration between Motherboard and LadyBits. Follow LadyBits on Twitter and Facebook.

Lead image: Lia Kantrowitz