A Computer Is Learning Common Sense By Trawling Pictures on the Internet

They say a picture is worth a thousand words, so perhaps it's no surprise that visual learning is the next big thing for artificial intelligence. There are billions of images and videos populating the web, and think about how much information is stored in even a simple shot of, say, a Kansas farm in the summer. Unfortunately, it would take humans a really long time to tag all those images in order for that data to be mined by today’s machine-learning algorithms, so it hasn't been helping computers get any smarter.

Researchers at Carnegie Mellon University have a new solution: a computer program called Never-Ending Image Learning or, charmingly, NEIL. The computer can make sense of images on its own with very minimal human interaction, just by looking at thousands of photos a day around the clock, and making associations between objects—eventually learning to draw its own conclusions about things that it was never programmed to know. In other words, it's developing basic common sense.

Here’s a theoretical example. If NEIL looks at thousands of photos of cats, and millions of colorful photos, it can use computer vision to recognize the shape of a cat and deduce the colors of the rainbow. Then, it can use those disparate pieces of information to connect the dots and rationalize that, since it never made an association between the color pink and image of a cat, there's no such thing as a pink cat.

Essentially the program, funded by Google and the Department of Defense’s Office of Naval Research, is mimicking the common sense humans form just by living life in the visual world every minute, day after day. Everything we do and see feeds into our well of knowledge and assumptions about the world. To borrow the example that MIT's AI expert Catherine Havasi gave the Associated Press, if someone asks us if a giraffe can fit in a car, we know the answer—not by calculating the respective size of both objects, but from that always-growing database of knowledge.

The principle behind NEIL is the same, and a recently launched website for the project lets you search through the database of images and knowledge that NEIL has deduced so far. Since the program started in July, it's identified roughly 1,500 objects and 1,200 scenes from millions of photos. From there, it's made 2,500 associations.

But, the associations aren’t always right.  A search for "BMX" on the website, for example, displayed a bunch of images of the bikes—so far so good—and a handful of relationships the computer determined. These were more iffy. NEIL determined that "Bicycle can be a kind of/look similar to BMX," but also "Umbrella can be a part of BMX." It also gets tripped up by homonyms—a search for "apple" will show images of the fruit and the computer logo. Humans would have to intervene to tell the computer it's incorrect if it tries to draw a relationship between an Apple laptop and a banana.

Researchers want to use the technology to build a massive database of visual information—it’s easy to imagine why this interests Google. They also hope that in the future, NEIL will be able to accurately answer common sense questions and make rational decisions—something the Defense Department has a keen interest in, so as to get more brain power on the battlefield without putting humans at risk. DARPA is currently working to develop computers that can not just think on their own, but can think on the fly, in real-time, by understanding the environment around them.

While algorithms and mathematical models that mimic the brain can make more accurate predictions and even diagnoses than a human ever could, a computer is missing the learned judgment or intuition of a human being. Even though that judgment is often flawed, most experts agree the decision-making process isn't complete without it—especially in matters of life and death. Programs like NEIL are a step closer to a future where man and machine work together, pooling their respective strengths to reach a super-human level of intelligence.