Sorry Internet, a Machine Didn't Actually 'Pass' the Turing Test

A computer posing as a 13-year-old Ukrainian boy isn't going to enslave us any time soon.

09 June 2014, 11:25am

If you believe the Press Association, a major milestone in the rise of the machines was passed this weekend. The basic gist: “A ‘super computer’ has duped humans into thinking it was a 13-year-old boy to become the first machine to pass the Turing test, experts have said.” Of course, other outlets have interpreted this news in their own enthusiastic ways; the Mail, for instance, dialled it up a level, insisting the result showed that the computer "can 'think'."

But what really happened? A computer programme called Eugene Goostman took part in an event at the Royal Society in London, held on the 60th anniversary of Turing’s death, in which five machine intelligences held text-based conversations with a panel of 30 judges with the intention of tricking them into believing they were human. The idea of the challenge, famously devised by Alan Turing in 1950, is that if a machine can have a meaningful conversation with a human, it suggests very strongly that it may be sentient – that it can "think" on some level.

The Goostman software posed as a young Ukrainian boy with somewhat-broken English, successfully convincing one in three judges – or 33 percent. On this basis, the organiser – cybernetics showman Professor Kevin Warwick – declared that they had passed a threshold of 30 percent required to "win" the event, opening up a new era in artificial intelligence. “This milestone will go down in history as one of the most exciting,” he said.

It does seem exciting, yes, but there is a serious problem at play here: the machine didn't actually pass Turing’s test. An AI is said to pass the Turing Test if it can reliably fool human interrogators. As Turing put it in his original paper:

“We now ask the question, ‘What will happen when a machine takes the part of [the test subject] in this game?’ Will the interrogator decide wrongly as often when the game is played like this, as he does when the game is played between a man and a woman? These questions replace our original, ‘Can machines think?’“

The key words here are "reliably" and "often". Turing didn’t ask whether a machine could sometimes, on certain occasions, convince a human judge that it too is human; he asked whether a machine could do so reliably, scoring at least as well as humans on average. In this instance, only a third of the judges were fooled – an impressive achievement, sure, but not exactly the kind of robust, repeatable science described in Turing’s paper.

Another problem is how the judge was fooled. In this respect, the approach Eugene Goostman’s developers took is brilliant – a textbook case of using lateral thinking to exploit loopholes and ambiguities in the rules of a contest.

The program itself is fairly standard stuff, a type of chatterbot that uses some combination of language processing, keyword matching and large databases of text to choose appropriate responses to text input. Chatterbots can do some pretty cool and interesting things – Apple’s Siri is one example – but they don’t generally fool humans.

To win this contest, Goostman’s developers added one key ingredient. Rather than try to beat the challenge head on, they tricked a judge into making allowances for their software by posing as “a 13-year-old boy from Odessa, Ukraine, with a father who works as a gynaecologist, and who owns a pet guinea pig”. Obvious fuck-ups that would normally have given the game up in seconds – nonsense sentences or inappropriate replies to questions – could then be explained by the "boy’s" poor English skills or youth.

It’s clever, and hats off to them, but I doubt it's what Alan Turing had in mind when he devised the test, and it raises all sorts of questions about the rules. Is there a minimum age limit for AIs taking part? Why not an eight-year-old, or a toddler? What about someone with the English skills of a recently discovered Brazilian tribesman? Not to deny Eugene its achievement, of course – as an exercise in fooling humans, Goostman is fascinating – but it has no real understanding of what it’s saying. It’s a superb piece of engineering, but it isn’t a machine that can think.

All this makes you wonder how useful modern day versions of the Turing test have actually been in driving artificial intelligence research in the last few decades. Researchers in machine-learning often talk about "Strong AI" versus "Weak AI". Strong AI is what you’d imagine; a sentient machine, general in purpose and knowledge – think Data from Star Trek, or HAL from 2001, or The Machine from Person of Interest. In contrast, Weak AI is more narrow; it has no real intelligence or awareness and relies on fairly specific tricks and techniques to solve a particular problem – think Siri, or predictive texting, or Google’s news clustering algorithms.

Turing devised his test with Strong AI in mind. He believed that sentience and information integration in some sort of "conscious" mind would be necessary for a computer to achieve a meaningful dialogue with a human, and that this mind would need to be connected to some way of experiencing the world – perhaps through a mechanical body: “In the process of trying to imitate an adult human mind, we are bound to think a good deal about the process which has brought it to the state that it is in.”

In contrast, Weak AI has dominated the modern incarnations of his test, and many of the competitors have been little more than browser games – chatterbots designed and engineered specifically to try to pass a pretty low threshold. Eugene Goostman is the strongest competitor to date, and a fantastic bit of work, but I’m sure the developers would be the first to admit that it has little value in terms of Strong AI research, while commercially it’s less interesting than more focused applications, such as Siri.

Arguably, researchers have made a lot more progress pursuing other challenges. IBM’s Watson super-computer, which famously beat human contestants at Jeopardy, is still very much a Weak AI – although, with something like 80 teraFLOPS of processing power, you wouldn’t say it to his face. It's also far in advance of anything taking part in Turing tests in terms of its ability to integrate information and extract meaning from it, one of the key measures of sentience. Watson may not be able to chat as fluently as Goostman, but it is capable of understanding far more.

Perhaps the most important thing we’ve learned from this is that achieving a 33 percent score on the Turing test isn’t as big a deal as we thought it was, and doesn’t bring us any closer to birthing our future machine overlords. Decades from now, a machine will arrive that can beat Turing’s challenge reliably, and that day will be mildly terrifying. For now, though, I’m not convinced that contests like Professor Warwick’s – with generous terms and a focus on quick wins and PR – are going to bring that day any closer.


Previously: Don't Blame Slender Man for the Schoolgirl Stabbing

Vice Channels