3D cameras have ushered in a new era of assistive technologies for people with disabilities. One such project, the Wheelie, allows users to control a wheelchair using only their facial expressions. Designed by Paulo Gurgel Pinheiro -- a Brazil-based Intel® Innovator, co-founder and CEO of HOOBOX Robotics -- Wheelie could radically improve the lives of millions whose mobility has been compromised by ALS, quadriplegia, and stroke. Motherboard spoke with Pinheiro about the promise of 3D cameras, and its potential to help us "hack disability."
Can you give me a little background on yourself?
I got my PhD in computer science and robotics in 2013. My focus was on developing localization solutions for mobile robots. You know Roomba, the robots that clean the floor autonomously? At that time, when I was in school, they were not able to go from point A to point B on their own. We built a solution for this. Afterwards, I started a postdoc program, which is when I began working with wheelchairs. We put that same localization system into wheelchairs, basically.
What do you mean by "localization"?
In order for a robot to go from point A to B autonomously, it first has to know where it is to start with. And it has to locate itself very quickly, because Roomba is supposed to clean the floor, not spend forever trying to find its poles.
This is the problem we solved. With Roomba, we installed a very cheap, tiny camera, which uses AI to match what it sees with a map of the house. But with the wheelchair, we have an Intel® RealSense™ 3D camera, which allows us to do the localization really fast.
Localization is one thing we can do, but it's not really the heart of Wheelie.
Wheelie is the first computer program that can translate facial expressions into commands that can be sent to a wheelchair, allowing a disabled person to control its movements.
We place an Intel® RealSense™ 3D camera in front of the user, which records their facial expressions; so when the user smiles, for instance, the camera records certain points around the mouth. Our software then classifies those points, recognizing that the user is performing a smile, and then translates that smile into a command for the wheelchair. So when the user smiles, the wheelchair will stop, for example. This entire process occurs in less than 0.7 seconds. It's just not something you can do with 2D cameras.
When someone uses it for the first time, do you have to calibrate it so the program recognizes that person's particular way of smiling?
No, it understands that the mathematics behind your smile is the same as my smile, even though we look different. This is the artificial intelligence in our classifier, which behaves like us. For example, I may never have seen a person before, but I'm able to recognize when a person is raising their eyebrows or giving a kiss. Wheelie uses the same behavior to interact with people.
What kind of hardware and software are you using?
For the camera, we use an Intel® RealSense™ F200. Recently, Intel demoed the Euclid, which is a wireless RealSense camera with a self-contained PC, all about the size of a candy bar. I'm very excited to get one next year and start running Wheelie with it. Actually, Wheelie could be a feature inside Euclid, which people with disabilities could use to control anything at home with facial expressions.
The set-up also requires an onboard computer. The one we use is an Intel® NUC, a mini PC that fits on the back of the wheelchair. The next step this year will be to get Wheelie running on the Intel® RealSense™ Robotic Development Kit, which is a smaller PC for RealSense cameras. We hope to do it this year.
For the software, we use the Intel® RealSense™ SDK to get those points on the face -- 78 in total. Even though the SDK comes with some programs that recognize facial expressions, we decided to build our own classifier to get that 0.7 second response time.
Is there particular significance to 0.7 seconds?
Yes. We can actually do it faster, but it can't be too fast because users need time to make an expression and then observe the wheelchair responding. We've found that 0.7 seconds feels natural and comfortable to most people.
How important has user feedback been to designing it?
When I started my postdoc program, I had to develop an interface to control a wheelchair. So we tried a lot of things, like putting sensors on the user's face, or goggles with cameras that track eye movement -- anything to make a wheelchair move. But when we talked to people with ALS, stroke victims or quadriplegics, many said they don't want to use any body sensors. First, they would need someone to put the sensors on every time. Second, it's not comfortable. And third, you can't go in public with a lot of sensors on your face and neck and head.
But we didn't have another solution. Moving around your home or office requires some delicate and complex maneuvers. Without body sensors, our program was just too inefficient.
Then I started working with Intel® RealSense™ last July. I built a very good classifier to use with it, which became the solution. Without using any body sensors, we have maintained -- improved, actually -- the efficiency of the program.
What was the biggest challenge you ran into?
To classify facial expressions. That was our main challenge. Especially tricky expressions like a half-smile. People have different abilities, of course; I can perform a smile comfortably, but for someone who has suffered a stroke, for example, it's not so easy. We work with some people with ALS, for instance, who cannot raise their eyebrows. So we had to improve the classifier to recognize other key expressions, like half-smile. Half-smile works great for stroke victims; it's one of the favorite facial expressions people use.
Every time we talk to a user, we get ideas about new expressions we could put in our classifier. So far, we have up to 7. You can choose the expression you want to assign to each command. Kiss to move forward, raise eyebrows to turn right, smile to stop, etc. But if you can't perform that many expressions, it's ok, too; we designed a circular menu with highlighting buttons that requires just two facial expressions to use.
Another challenge was to make it possible to use Wheelie in different lighting conditions. We want to to use it in a bright room with a lot of light, but in a dark room, it's supposed to work as well. It took a lot of work to do this, as well as put it all together in a comfortable interface.
How do you know when the user is smiling at a joke or smiling to control the chair?
First off, we have voice recognition that allows them to enable and disable the interface anytime they want, so they can react normally without interfering with the system. But to answer your question, we can't tell when a person is smiling to control the wheelchair or smiling at a joke they heard. That's why we always use smile to stop the wheelchair: for safety.
We can tell when a person is coughing or sneezing, though. We have a classifier for this, so it won't interfere with the system. It's so elegant. We know when people are going to cough before they do it, but smiling is trickier.
How do you see this technology improving lives of people with disabilities?
There are more than 3 million people in the world that would benefit from the Wheelie: people with ALS, quadriplegia, victims of stroke, etc. Every year, more than 400,000 people worldwide are going to suffer an injury, many of whom could benefit from the Wheelie. What we started to do here was to develop a mobility solution. But now we know that Wheelie is more than that: The users are not only improving their mobility, but also their self-esteem and their autonomy. This has become more important to us. It's a good thing we did not expect to happen.
3D cameras are able to interact with people in a very natural and comfortable way. Using this technology, I think we have a chance to, as I like to say, "hack disability," just like one hacks a computer to install software to make it run better. If I can provide a Wheelie to improve a person's mobility, and then this user can interact more with people, decreasing their isolation, closing that social gap between them and others, I'll be happy.
The Intel® Developer Zone offers tools and how-to information to enable cross-platform app development through platform and technology information, code samples, and peer expertise in order to help developers innovate and succeed. Join communities for the Internet of Things, Intel® RealSense™ Technology, Modern Code, Machine Learning and Game Dev to download tools, access dev kits, share ideas with like-minded developers, and participate in hackathons, contests, roadshows, and local events.