You usually control virtual reality games with a gamepad, gestures, or the movement of the headset itself. But a new tech demo imagines a future in which the player instead interacts with the world using speech, and it’s pretty compelling.
In Garden Friends, which overlays 3D characters on the real world, a Pokémon-looking character known as the “lonely gardener” provides the player with seeds that grow into trees, flowers and vines. Say nice things to the plants—“you’re a beautiful and extraordinary tree”—and they blossom and grow; say mean-spirited things, and they wither under a black cloud.
“Currently, most AR interactions are just based on tapping on your device, or ‘approaching’ a character in AR space,” said programmer Nicole He, one of the creators of the demo and a creative technologist at Google Creative Lab. “But using speech recognition as a way to control a scene in AR is something that's still pretty new and experimental, which is what we were playing with here.”
He created Garden Friends with artist Eran Hilleli, with technical input from AR developer Anthony Marefat. They used the Unity game engine, the ARCore SDK for the augmented reality component, and Dialogflow for speech analysis.
The game isn’t currently publicly available, but He and Hilleli may release it on the Android Play Store.