This story is over 5 years old.


How Tomorrow's Battles Will Be Fought in Augmented Reality

Sophisticated technologies may optimise a soldier’s situational awareness, and ply him or her with information.
The military commander of the future will be making decisions based on the visuals from multiple datasets. Image: University of Birmingham/BAE Systems

Imagine a battleship's command room—but instead of a physical display screen, there's a levitating 3D dome. It displays everything that's going on underwater and in the air, providing military commanders with an unparalleled view of the battlefield. Everything in this microcosm is actually happening in real-time, and it's how Bob Stone thinks the future of war might be fought.

Stone's musings on the future of how we monitor and conduct warfare almost sounds like an infinitely upgraded and sleeker version of Warhammer, an old-school tabletop wargame featuring miniature fantasy regiments. And though what he imagines might not be reality just yet, since 2013, he's been collaborating with engineers at BAE Systems—a British multinational defence, security, and aerospace company—on mixed reality systems that meld the real and virtual worlds.


The project's aim is to create augmented reality systems that optimise the situational awareness and decision-making processes of military personnel of the future. But questions of how to do so safely, effectively and accurately remain to be answered.

"Does it need to be visual? Does it need to look like the real world? What happens if we introduce video, and live feeds from news channels and social networking sites?" asked Stone. "Nobody knows how much information is too much just yet."

Just a mirage? An avatar stands in front of a virtual display. Image: University of Birmingham/BAE Systems

Stone has been working on virtual reality systems for over two decades. As an applied psychologist by qualification, he is interested in evaluating what kind of information military personnel need to make sharper decisions that would lead to less casualties in the future.

"I'm waiting for the day when someone produces a very large volumetric display—there's nothing like that yet," he said.

One current idea that would allow soldiers to adapt more quickly to their environment is a "Wearable Cockpit." Headset clad pilots could customise their aircraft's interface through reconfigurable virtual displays and controls based on personal preference and mission objectives.

Another idea is a "Portable Command Centre." According to Stone, a user wearing a virtual reality headset and interactive gloves might control a mixed reality environment—with virtual displays for landscape, newsfeeds, data from remote UAVs and command centres—that seems to appear in front of them.


A researcher testing out tech in Bob Stone's lab. Image: University of Birmingham/BAE Systems

While some of these ideas are already being explored as virtual reality headsets become cheaper and less expensive to produce, what remains largely unexplored is the long-term effects such experiences might have on the human body and the mind.

Stone, for example, gives questionnaires to armed forces personnel who trial his tech in order to gauge their feelings and observations on how the tech could be improved and adapted to suit their needs. He measures galvanic skin responses (the same kind of tech used in lie detectors), which he said "detects how stressed someone can be," and also intends to measure brain function by using low-cost brain wave sensors.

As battlefields of the future become populated with autonomous vehicles and robots, as well as headset-clad soldiers with the ability to see through buildings, draw up virtual display screens, monitor news and social media feeds in real time, and fight battles remotely, having access to accurate information will also be crucial in making the correct decisions.

Sophisticated mixed reality technologies may optimise a soldier's situational awareness, and ply him or her with more information. Yet, conversely, Stone acknowledged that the representation of uncertainty was still a "thorny question" that might inhibit a soldier from making a balanced decision.

The future of surgery? Augmented reality mashed with physical displays and remote-controlled surgery robots. Image: University of Birmingham/BAE Systems

"Imagine there's an unmanned vehicle, and it's flying around an area and we lose transmission for half an hour, but some transmissions are still coming back," explained Stone. "How can we represent how the probability of that information is accurate?" The possibility of information degradation would definitely have to be taken into account.

In the meantime, Stone told me that his augmented reality systems were not merely limited to the military sphere. He is currently in discussions with forensics specialists and surgeons too. And while the prospect of a 3D bubble containing a microcosm of the battlefield of the future doesn't exist yet, Stone remains excited by the prospect of what current and upcoming virtual reality technology can do.

"After being involved in VR for 28 years, it seems like the applications are finally coming to the technology, and that's good news," said Stone.

All Fronts is a series about technology and forever war. Follow along here.