Projector Jewellery Turns Your Hand into a Touchscreen

I’ve never felt that taking my phone out of a pocket to look at it is a hugely time-consuming or irritating action, but wearable tech keeps trying to get around this everyday motion by putting information where we can immediately see it. There’s smart watches, Google Glass, and now a jewellery-mounted projector that turns your hand into a touchscreen and casts your messages onto the floor.

New Scientist reported on the “Ambient Mobile Pervasive Display” developed by a team at the University of Ulm in Germany, which will be presented in a Toronto conference in April. It’s an LED projector system intended to be worn in a necklace or brooch that projects an information display in front of the wearer—onto the floor, and then their hand.

The researchers explain on their website that you use hand gestures to interact with the technology: “The display is combined with a projected display on the user’s hand, forming a continuous interaction space that is controlled by hand gestures.” For instance, a text message appears as a projected shape on the floor. You "pick it up" and the sender's name appears on your outstretched palm. You then make a gesture that opens the message, and read it on your hand. So basically, instead of holding a device with a touchscreen, you kind of pretend your hand is a touchscreen (which makes me wonder, is that how kids play telephone these days?).

In New Scientist’s video above, you can see a prototype of the technology in action (at this point it's not embedded in a piece of jewellery but in a piece of rather clumsy shoulder-mounted apparatus). The user reads a text message on his hand; is guided to his destination by a glowing line on the floor; and is reminded to follow his fitness regime by a light-up sign that tells him to take the stairs.

Perhaps what’s most interesting, however, is when the projection system interacts with external “public” information rather than emails and texts conveyed directly from the user’s personal device. In a video of their own (below), the researchers show off this aspect, which is where the floor display really comes into play: when the user comes across a projected shape, they can interact with it. For example, another person could leave a photo lying on the floor, or a note in front of their office door, which you could then “pick up.” Perhaps inevitably, one example they suggest is targeted advertising: you could find ads projected outside shops that display personalised offers when you interact with them.

“All digital content is fixed to a physical location in the world,” the video narrator explains. “Whenever the mobile projected screen of the user overlaps this location, the user can see the item’s corresponding content. Users can interact with objects in this world with gestures in the air.” It’s a pretty cool gestural interface; you essentially scoop up the light from the floor and its content—a message or image—materialises in your hand. You can then drop it back on the floor, or throw it away over your shoulder.

The whole floor-to-hand projector system is made possible by a 3D sensor that can tell how far away your hand and the floor is, and so can refocus the information to make it readable. In a paper outlining the project, the researchers explain, “Floor and hand tracking is computed on the depth image from the camera. On every cycle, the algorithm first decides whether the user’s hand is present in the depth image.”

So far, so Minority Report, but the system has some obvious limits. It’s in its early days—the team said a consumer version could be feasible in two years—so some of these could be ironed out, but some seem to be a result of the base design. For a start, there’s only so much you can do with a hand-as-a-screen. The messages are necessarily in large text, so reading anything longer than a brief text would be a trial. And to answer, you’d still have to dig out your smartphone. In the videos, it’s also clear that the user has to look at the floor a lot, which perhaps isn’t ideal for going about real-life business. And the floor projection is naturally limited in size, so when the user follows directions cast in front of him, for instance, it looks like he can only see a few steps ahead.

The researchers have tested their prototype system on six untrained users and concluded that they had a “generally very positive attitude” towards it. They now plan to conduct a larger user study in public places, as that brings its own issues. They reported in the paper, “Two participants were concerned with performing large, eye-catching gestures in public space.”

In a future where we’re all using projected pervasive displays, I can also imagine the rather unappealing scenario of the floor being littered with random public information everywhere (like having everyone’s Instagram feeds in your periphery at once) and people accidentally hitting each other as they attempt to discard the content they’re not interested in. But then I never imagined a future in which brooches could make a comeback, either.