Edit Reality by Virtually Controlling Physical Objects

A new app from the MIT Media Lab’s Fluid Interface’s group allows you to connect real objects to the virtual world.

by DJ Pangburn
Dec 10 2015, 4:10pm

Courtesy of MIT’s Fluid Interfaces group

Technologists are busy creating and selling the idea of the Internet of Things: a network of physical objects that collect and exchange data, all controllable by the user. The only problem is this data would be owned by the companies that sell products plugged into the Internet of Things.

But researchers at the MIT Media Lab’s Fluid Interfaces group, led by Valentin Heun, are taking another angle. They reject the idea that the data produced by users’ products should be owned and stored by anyone but the user. So they’ve produced an app, Reality Editor, that allows users to virtually control physical objects by creating connections between them, all while retaining privacy and ownership of the data.

Courtesy of MIT’s Fluid Interfaces group

Heun envisions something he calls “Connected Objects” instead of an Internet of Things, controlled by the Reality Editor, which communicates with the world via standard networking technologies like wifi. The user simply points a mobile device’s camera at an object and its invisible capabilities will be made visible for editing. The objects send their “FingerPrints” (patterned stickers) and network IP to Reality Editor, and then users can drag virtual lines from one object to another, creating new relationships between objects.

Instead of being always connected to central entities, objects built for the Reality Editor—built on HTML5 and Javascript—live in decentralized private networks. And Reality Editor would never need a central cloud service to work, since every interface, data connection and type of functionality is saved in the object itself.

Courtesy of MIT’s Fluid Interfaces group

As Heun demonstrates in the video (watch below), a person could attach FingerPrints to a control knob, a light and a chair. With Reality Editor, the person could then create virtual relationships between the three objects, allowing the user to adjust the light’s intensity as well as the height of the chair. Heun also uses the Reality Editor to virtually connect a car’s windows and air conditioning system to knob. When the knob is turned to a certain value, the windows roll down and the air conditioning shuts off. And these are just a couple of the current use cases. Heun also envisions more elaborate scenarios in an “imaginary future existence.”

In one, a person buys an old radio flea market. With Reality Editor, he connects the radio’s knob to his smartphone’s “Hello World” calendar which awakens him. When he whispers “Good Morning World,” it activates a little toy that his children almost threw away—a little projector that shows the time in colors that can be altered by rotating a wheel. This color setup is connected through Reality Editor to the red, green, and blue values of a rooftop color sensor, which now displays the morning sunrise in his room through the toy projector. When he rotates the toy’s wheel some more, the window blinds open via virtual connection with the Reality Editor, which activates his coffee machine and awakens his car.

Courtesy of MIT’s Fluid Interfaces group

It’s like the Rube Goldberg morning scene in Pee Wee’s Big Adventure, but networked between the virtual and the real. Though again, all of these virtual-to-real connections, all of the data it produces, would be owned by the user. Everything would be private, except for when it was absolutely necessary to connect to a third party’s network, such as a car’s computer.

“If I switch a light switch on at home that light switch communicates directly with the light,” Heun tells The Creators Project. “There is no need to send this action all around the world and then back into my home. Data connections should always take the most possible direct rout, reflecting a user’s privacy interests.”

Courtesy of MIT’s Fluid Interfaces group

“I see it like this: a thing is something we own and we have a private interaction with. We strongly believe that when we buy something, we own it,” Heun adds. “And when we connect our things to the internet, we wish that ownership and privacy would be honored. I am not sure if there is a single Internet of Thing product that reflects such thinking. This is the reason why I like to talk about Connected Objects instead of the Internet of Things.”

The interesting thing about Reality Editor, as Heun explains, is that it would give new life to old objects—things like radios or other modules that have knobs and are still interesting. And in the future, Heun believes that users might buy new modules that represent individual functionalities instead of finished product entities. A user could, for instance, buy a two-rotation knob module that perfectly fits their personal style, and then combine it with a radio or streaming module and a speaker or screen.

REALITY EDITOR from Fluid Interfaces on Vimeo.

Eventually, Heun plans to fully develop Reality Editor’s 3D object recognition capabilities. This would reduce the need for the FingerPrints. These and other technological advances on the horizon will further blur the boundaries between the real and the virtual.

“Looking at history of computation and what is ahead of us, we can consider it a fact that within 10-20 years we will not be able to define such boundaries anymore,” Heun says. “What we consider virtual or digital is defined by the paradigms of a personal computing. The Reality Editor is a first small step to create a new set of paradigms.”

Reality Editor can be downloaded now from the iOS App Store, and users can use the Fluid Interface’s open-source Open Hybrid platform to build “hybrid objects.”


Man vs. Smart House: A Cautionary Tale

What Is the Internet of Things? An Interactive Tool Lets You Find Out for Yourself

Six Unique Prototypes On The Fringe Of The Internet Of Things