This Computer Painting Program Has Feelings

Imagine if Photoshop had feelings and helped you only when it wanted. If it was in a bad mood—say, after reading war coverage—it would refuse to paint. Try again later.

That is a bit like the virtual painting program, The Painting Fool, created by computational creativity professor Simon Colton at Goldsmiths in London. The Painting Fool has painted 100 portraits without human instruction for the Emotionally Aware Portraiture series at the Galerie Oberkampf in Paris. The exhibition, You Can’t Know My Mind, was set up like a modern-day photo booth—gallery-goers could sit down and get their digital portrait in the mood of The Painting Fool. Armed with emotion-detection software, it yielded some surprising results. It was supposed to paint people's emotions; instead it painted with its own feelings. Some portraits were bleached out and gray when the software was depressed, others had bright swirls when the software was happy. The Painting Fool can read news and emotionally react. It also measures each portrait’s appeal. “I was in an experimental mood. So I wanted to paint a hot portrait,” it has said. “This is a miserable failure—I'm very unhappy about that.”

While The Painting Fool has its mood swings from time to time, it does not require any kind of therapy just yet. It is just another artist, after all, and we all have our ups and downs. Colton remains enthusiastic about its growth and the future. I gave Colton a call, and we talked a little bit more about engineering creativity, the humanity gap, and computer coding a palette of six emotions into a multitude of art.

VICE: What is emotion-detection software?
Simon Colton:Emotion-detection software is able to take a photograph, or better, a video clip of someone expressing an emotion, and determine with some level of accuracy which emotion they are expressing. The emotions are often restricted to one of six types: anger, disgust, fear, happiness, sadness, and surprise. I worked with vision experts Maja Pantic and Michel Valstar at Imperial College to get their emotion-detection software to work with The Painting Fool so The Painting Fool could paint portraits which matched the sitters’ emotions.

The Painting Fool is actually quite smart. While it used to read the subject’s emotions, it recently began developing its own moods, quirks and personal habits?
Yes, that’s right. In the earlier project, when people expressed emotions for the software, they were given the choice of what to express, and hence they controlled the software to some extent—if they smiled, for instance, The Painting Fool would paint in a particular style. In the later project, the software itself simulated moods with methods relying on sentiment analysis of newspaper articles. This brought about a flip: the software needed to control the people, so that it could use them as a model to express its own mood. That is, if it was in a bad mood, it told people to look sad, so that the portrait would ultimately express The Painting Fool’s (simulated) mood, rather than the mood of the sitter.

It isn’t like Photoshop, the software here has its own imagination. What kind of surprising things does it do?
We actively write software so that we don’t know in advance exactly what it is going to do. We put together collage generation software, which produces paintings to illustrate newspaper articles. One morning, the software ran (on a schedule—without us asking it to) and read the headline article, which was about the war in Iraq. We had no idea this would be that morning’s subject matter, we had no idea which keywords it would extract, and we had no idea which images it would put into the collage—so there was much opportunity for surprise, or at least unpredictability. The collage it produced that morning was stunning: containing images of fighter planes and explosions, with a picture of mother and baby in the center, and a girl in local headgear too. Most poignant of all, one of the images in the collage was a field of war graves—this really made the hairs on the back of my neck stand up! As an illustration of the horrors of the war in Iraq, this image was surprisingly good.

How much does the Painting Fool work independently of human instruction?
This depends on the project. When I’m producing artwork for a gallery exhibition, I have very high standards, so I undertake a lot more production and curation work. In other projects, the whole point is not to interfere with the software. For instance, for a week-long demonstration in Paris this year (as part of the You Can’t Know My Mind exhibition), the Painting Fool produced portraits entirely independently of me or anyone else. Over that period, it learned to get better on its own, it expressed moods and intentions, painting around 100 portraits.

So it not only paints but gives feedback to the model on its current mood. How often is it in a bad mood versus good?
For the You Can’t Know My Mind exhibition, The Painting was in one of six moods: very happy, happy, experimental, reflexive, sad, and very sad. It was in each of these moods for roughly the same amount of time, but we had no idea at any time which mood it would be in—we couldn’t know its mind, because that depended on where it had recently been in terms of reading the newspaper articles. When in the very sad mood, it told people to go away! That is, it didn’t feel that it was appropriate to paint a portrait, so it asked people to come back in ten minutes. While this was funny and occasionally frustrating, it emphasized the point that the software is not commanded by the sitter, me, or anyone else—it was acting independently. When it sent people away, it gave them an explanation of its decision in terms of what it was recently reading, along with some of the particularly negative key phrases that it had extracted. It ended the explanation by stating: “No random numbers were used in coming to this decision,” which was really important. This was not a chance, arbitrary decision, but based on intelligent understanding of newspaper articles.

It has been known to ask people to smile, what else?
In a bad mood, it asks them to think about something really sad, and look very serious. In an experimental mood, it asks them to pull a weird face, look away from the camera, or in general do something unusual. One sitter replaced his face with his hand—and the portrait was fine!

How exactly does the Painting Fool display its own feelings in each portrait?
In displaying its own mood, The Painting Fool chooses color palettes, muted colors for sad, bright colors for happy, image filters, and abstract art for backdrops. It knows which pieces of abstract art of a million possibilities people are likely to project certain adjectives onto. For instance, with a really red image, it’s likely that people will project the word “fiery” onto it, but for a really blue image, the “cold” is more likely. It can then choose an adjective according to its mood, that is, bright, happy, colorful if it’s in a good mood, or bleary, dull, grey if it’s in a bad mood, and so on. Then ask the [subject] to express a suitable emotion, choose the backdrop abstract image as best as possible to express that adjective, and finally choose a medium and painting style to emphasize the mood, which is charcoal and pencils for dull or grey images, and bright acrylic paints for happy and colorful images.

Why is it based on emotions? Why did you want to bring that out in people? I know some artists who work methodically without being inspired because, while creative, it’s still their job.
I’ve talked in the past about “the humanity gap”—that software isn’t human, never will be human, and that we should celebrate this, not be ashamed of it. The humanity gap is very real, and is why some people will never like software-generated artifacts like poems or paintings, because they want to gain a human connection with the artist/poet, which is perfectly valid. To begin to fill the gap, we have to show that the software exhibits certain intelligent behaviors associated with creativity, such as: skill, appreciation, imagination, learning, intention, and reflection. In addition, while not pretending that software right now has emotions like people do, we can show that it can appreciate emotions in people, and can use accountable simulations of mood to drive its painting process. We hope that these things help people to appreciate the value of The Painting Fool as an artist more than they would have otherwise.

The goal has been for the Painting Fool to be accepted as an artist in its own right. How much of this have you achieved?
Well, nobody is yet claiming that The Painting Fool is a van Gogh or Picasso! However, at the You Can’t Know my Mind exhibition, both the quality of the artworks on the walls (all produced by The Painting Fool), and seeing the software paint portraits completely independently, went a long way to giving people the impression that the software was a painter. Now, this doesn’t mean that people think the automated painter is particularly artistic (there is still a lot to teach it about the art of painting) or particularly creative (in most pathways, it doesn’t come up with any new ideas, for example – but we are working on that). Hence, we certainly haven’t achieved the entire aim yet, but with this year’s exhibition, we went a long way down the path.

@nadjasayej

More news about robots:

This Machine Frees Oppressed Chickens

Did Robotraders Know the Financial Crisis Was Coming?

Israel's Killer Robots