Artist Sebastian Schmieg On His Time-Lapse Expressway Into Google Search's Soul

One might imagine in a few hundred years time that the Google search algorithm with be regarded as a deity. Users will make a little gesticulation in the shape of the letter “g” before hitting the “I’m feeling lucky” button, which will, by then, be the only way to Google search because asking for a list of results from the almighty is to show a lack of confindence in It. It’s strange that this extremely powerful thing — you know, the gatekeeper to civilization’s knowledge etc. — is essentially a black box for most people, in some part because it’s kept secret by Google (the specific nuts and bolts anyhow) and in some part because it’s just kinda hard to understand. It’s math and people hate math (math is hard, especially this math).

German artist Sebastian Schmieg has devised a sort of backdoor to understanding, however, in his “Search By Image” project. You can look at a handful of said project’s iterations below, but the basic idea is to follow Google search into itself at a high rate of speed. Specifically, he searches based on one starting image with Google’s recently introduced “search by image” feature, searches the result of that first search, and keeps going. And then Schmieg spools it all up into video clips; equations then become something visceral.

There is a degree of understanding to be had in the results, of machine learning in general and what’s inside the black box of Google. You can feel how it happens, which is something beyond computational nuts and bolts. Over the weekend, I asked Schmieg about the project, his thoughts on search, and when a tool becomes something more.

What’s the root/genesis of this idea?

A friend told me about the new feature a few days after it was released. I was fascinated by the possibilities of searching not by words, but with an image. When searching with Google, one mostly finds what one is looking for without surprises. Using an image is a really new way of navigating the database through which Google tries to represent the web. And pretty quickly I realized that often strange things happen. A human face leads to a naked baby, to a naked woman etc. Or something like that. The algorithm tries to understand the image but sometimes it fails badly – well, depending on what you actually want to do with it.

When starting with my experiments, my intension was to playfully examine, misuse, and critique this new thing, which I think is way more than just ‘a tool’.

What are you revealing with these? What are some of the biggest insights into search you’ve gleaned from the project?

Each video tells a different story, I guess. The one video starting with a transparent png seems to tell a story about human evolution as represented by the images on the web. But keeping in mind that the starting point actually is some kind of ‘nothing’, it also becomes evident how Google’s search algorithms and database structures determine where you go. It is as much about allowing you to search for something, as it is about guiding you to find something.

The video that is starting with a found photo of the Earth explores two additional aspects: how one image exists as many copies, and versions on the web. And how Google’s tool allows copyright holders to track down those copies.

While trying to remake sequences of movies through the use of Google’s search by image, I was surprised to find out that the results looked so much more random than I had expected. Furthermore, through the YouPorn video, I learned that there’s many photos on the web that show people's diseased genitals.

In general, I think Google’s search by image is impressive, but also satisfyingly inaccurate. Soon, it is one year that the feature has been released, so I will compile a new set of videos. I am curious to see how it will evolve.

What are your biggest concerns with Google search and search in general?

Even though links on Facebook or Twitter are now often the starting point for surfing the web, I think that search engines still play a crucial role in structuring the web, and in guiding as trough it. And with new semantic ways of searching, they will probably strengthen their position again.

In general, I think it is important to examine algorithms that shape a big part of our online and offline reality. In the case of search engines, they have a big impact on our collective memory, and on my personal perception of the web. An archive is not only its content, but also that which is not part of it, and its set of rules. Therefore I am interested in playfully exposing algorithms (not their code), in making them more transparent, less mysterious. The strategy I currently find very promising is one of ‘performing’ algorithms, which in the best case leads to a story about their inner workings.

Can you talk about the process of doing these? How tedious is this? Do you have something automated to help out?

The whole process of making a video is automated. I only need to choose an image to start with, or a video sequence to redo, and a bunch of little scripts will do the rest. You can find the most important part of the code at the bottom of the project’s page.

Do you have any experience with search algorithms outside of this? Or, have you done much research into the actual mechanics/machine learning algorithms, such that it’s even possible?

No, I’m no expert in this field at all. I tried to find some information about the algorithms behind Google’s search by image, but as always, there isn’t much to find. In general, I have a vague understanding of computer vision that I have gained through projects in the field of interaction design.

If were able to ask a Google engineer anything and get a straight-up honest answer, what would it be?

What’s the salary of the guy typing in the auto-suggestions while I search?

• Creep Files: BP Buys Search Terms For Oil Spill
• New Search Engine Does Something Google Can’t
• Is Google Stereotyping the World?

Reach this writer at michaelb@motherboard.tv.