Does your phone know the difference between a hammer, a wine bottle, and a hot dog?
Probably not—but with a bit of coding, it could. Programmer Chris Greening connected an iPhone camera to CoreML, Apple's open-source machine learning framework used across products including Camera and Siri, with Apple's Vision Kit
Greening outlines the step-by-step project on his blog. To do this yourself, you'll need a device that's running iOS11 (which, he notes, is still in beta, so you'll want to tinker on a phone that's not essential to your everyday existence) and the beta version of Apple's xCode. He's published the entire coding process on Github.
It recognizes the iPhone as an Apple device, but is about 90% sure it's an iPod, or some kind of cellular phone, or a modem, or a speaker, which isn't… technically incorrect, is it? It nails it on the spatula and hammer, but a coffee mug is a little more uncertain—is it a measuring cup? Mixing bowl? Definitely something that holds liquids in the kitchen.
Not bad for a model doing all of the processing on-device, without using server calls. It's running ResNet50, a model developed by Microsoft researchers that includes 1000 categories such as trees, animals, vehicles, food and people, according to the CoreML website.
Get six of our favorite Motherboard stories every day by signing up for our newsletter .