This story is over 5 years old.


Here's What Actually Goes into Creating Artificial Intelligence

Artist Sebastian Schmieg visualizes the image outlines created by Amazon Mechanical Turk workers for Microsoft’s COCO project.
Images courtesy the artist

People could be forgiven for thinking the building of neural networks and artificial intelligence isn’t labor intensive. That these things are simply the work of creative engineers and computers alone. But as artist Sebastian Schmieg illustrates in his latest work, Segmentation.Network, there is actual human labor behind the artificial intelligence and neural networks. Not just humans dreaming up advanced algorithms, but a labor force tasked with teaching the algorithms how to think and dream.


As Schmieg explains, Segmentation.Network “makes visible the hidden and often exploitative manual labor that goes into building neural networks and artificial intelligence” by playing back over 600,000 segmentations (basically, an image outline) created by crowd workers for Microsoft's COCO image recognition dataset. The dataset is derived from photos on Flickr and is used in machine learning for training and testing purposes.

What Segmentation.Network plays back is a random collage of image segmentations at their original position. Each segmentation consists of multiple coordinates, and the piece shows the manual process of creating these coordinates in fast-forward. To create Segmentation.Network, Schmieg downloaded the whole dataset, extracted the segmentation data, and is now drawing it with Javascript.

“A segmentation is an exact outline of all objects deemed relevant inside a single image,” Schmieg tells The Creators Project. “Each outline is associated with an object category. Outlines are created manually click by click. Basically, it's the most detailed understanding of an image and its content that you can produce.”

Originally, Microsoft's research team selected and downloaded a total of 328,000 photos from Flickr using a combination of search terms. Workers in Amazon's Mechanical Turk platform then analyzed and annotated each image. In total, these workers spent more than 70,000 hours creating the dataset, according to a Microsoft paper.


“First, workers had to determine which object categories are present in an image,” Schmieg explains. “Afterwards, each object had to be labeled. And finally, an outline had to be drawn around each instance of an object. This is the data played back in Segmentation.Network.”

Schmieg’s interest in algorithms and databases goes back to the beginning of 2012, when he published a piece called Search By Image, which utilized Google's image search of the same name. Since then he has been artistically and critically working with algorithms and databases, asking how they shape online and offline realities. And, like many others, he has been fascinated by things like Google's deep dreaming experiments and, more generally, advances in artificial intelligence.

“Computers and neural networks are learning to see and even to hallucinate, offering a new kind of vision,” Schmieg says. “However, taking a closer look by reading research papers and reading through crowd work forums, it became obvious that this isn't the achievement of clever engineers and powerful computers alone. There is a huge amount of underpaid and anonymous ‘crowd workers’ preparing and optimizing datasets and algorithms.”

Without their work, Schmieg says that the “so-called artificial intelligence” wouldn't look as promising or frightening as it does today. These workers are distributed all over the world, and do this kind of work either full-time or as supplementary income. And they are not at all affiliated with Microsoft. Other image analysis projects include ImageNet (which doesn’t feature image segmentation), PASCAL, and SUN.


“Working primarily with technology I think it is important to look beyond shiny interfaces,” says Schmieg. “What often seems immaterial or almost invisible does always have very physical conditions in which it is being created, maintained and executed. In this case, the crowd workers deserve as much attention and appreciation as the engineers.”

“Teaching machines how to see is a very insightful and self-reflexive endeavor in which you have to start at the very beginning as computers are primarily a text-based medium,” Schmieg adds. “Basically, machines can only see what we teach them to see.”

This, Schmieg believes, allows those working on machine vision to depart from questions like, “What do we see and what value do we assign to what we see? What happens if we see something? What is irrelevant?” and so on. For him and others fascinated with neural networks and artificial intelligence, machine vision doesn't have to be only about surveillance, war at a distance, or targeted advertising. It can be about many other things—including creativity.

Click here to see more work by Sebastian Schmieg.


Obliterated Hard Drives Are Found Art for the 21st Century

A Look Inside the Academy Award-Winning 'Ex Machina'

A Robotic Ceiling Sets the Precedent for AI Architecture