How the Tech Behind 'Pokemon Go' Has Been Powering Games for Decades

There's an overwhelming sense of awe as you hike through the trails of Yoho National Park in the Canadian Rocky Mountains. From its soaring mountains to the emerald lake beneath, the scene is a feast for the senses.

It's here where Teri Rueb, an artist known for her location-based interactive works, created a sound installation called Trace. Arming hikers with a location-aware device in their backpacks, Trace overlaid an augmented audio reality (or augmented aurality for short) throughout the terrain. As hikers moved, their location shaped what they heard, with every twist and turn unearthing poems, songs, and stories about the land.

Trace falls under locative media, a type of interactive work where a user's location shapes how the work itself is experienced. Given the prevalence of smartphones with GPS-enabled capabilities, works like Trace aren't surprising nowadays. But Trace wasn't created in 2016, it was done in 1996.

While Pokemon Go may have brought mainstream attention to locative media last summer, this is not a new medium at all. The field has been maturing during the past two decades, and something like Pokemon Go is just a tiny glimpse of its potential.

In 1996, few people carried mobile phones, and fewer still had internet access on them. For early locative media artists like Teri Rueb, creating experiences like Trace required a healthy dose of ingenuity.

"I was using a laptop that was about ten pounds," says Rueb with a laugh. "You had to load both battery docks with batteries for it to last even eight hours. The batteries were really heavy…[and] I had an external GPS, another big chunky black box [all] bundled in a backpack."

_Trace Trailer​​_​

Rueb explains that back then, general consumer laptops weren't designed to withstand the power usage and heat that Trace generated over a long period of time. To counteract these issues, she recalls having to devise DIY workarounds, like placing foam wedges between the laptop screen and keyboard (to close the laptop but not fully so that Trace could still run) to sticking a couple of blue icepacks in each backpack to avoid overheating.

It was an elaborate system, but like a magic trick, it didn't matter how they did it, just that it worked. "People weren't meant to look in the backpack," says Rueb. "Most people were willing to just put a backpack on that weighed ten pounds and not think twice about it."

It's the same experience Matt Adams would see a few years later as part of Blast Theory, an artist group known for their interactive and live performance works. In 2001, with the help of the Mixed Reality Lab at the University of Nottingham, Blast Theory created one of the earliest location based games called Can You See Me Now ?

Can You See Me Now Trailer

It was a game of chase where players tried to avoid detection from Blast Theory runners. The caveat? Players were online and the "runners" played in the real world. While players used arrow keys to move their avatars inside a virtual city, runners had to physically run in the city streets. The runners' physical locations appeared as avatars inside the virtual world. Meanwhile, the player's location "showed up" in the real world via a map on the handheld computers runners used.

Just like Rueb, creating Can You See Me Now? required Blast Theory to fiddle around with homemade solutions.

"It was very hand-cranked and laborious. At that time, no one had broadband outside of universities [and] Wi-Fi networks themselves were pretty few and far between," recalls Adams. "We essentially [had to] set up our own Wi-Fi networks to do it. We would climb up church spires, tall office buildings, multi-story car parks and clamp Wi-Fi transmitters (which were kind of the size of a small golf bag)…to establish a network."

In those early days then, it was a technical feat just to record live data and report someone's position (as in Can You See Me Now?) or even activate audio recordings (as in Trace). Despite these challenges though, artists like Adams and Rueb were enamoured by the medium. "It was super exciting," reminisces Adams gleefully. "It was one of those things where you immediately could feel that this had potential."

A lot of that potential would be realized by creators who wanted to tell new stories. One transformative example was a 2009 interactive fiction piece penned by indie game developer Pippin Barr: Crisis 22. It's a horror story about a person walking down a city street, with the player literally walking down the street in order to progress. Unlike browser-based interactive fiction where we navigate through hyperlinks, Crisis 22 navigated the story via physical space.

As Barr describes it, space equalled time. Walk forward to your rendezvous point and an ordinary linear story unfolds. But, if at any point in your walk you diverge, different aspects of the story emerge. Walk backwards and you'll uncover the character's backstory. Stop and you'll get an internal narration of the character thinking through things. Turn right or left and you'll reveal peripheral events. As you veer off the normal route, the story becomes more surreal as you see disturbing, weird, and monster-like folks around you.

Crisis 22 photo courtesy of Brian Greenspan​
​

The piece was created as a tech demonstration for a location-based storytelling tool called StoryTrek. Created in collaboration with Barr's colleagues including Brian Greenspan, Rilla Khaled, Robert Biddle, and Chris Eaket, the initial prototype mapped narrative properties within the physical world.

Unlike waypoint based narratives though (where going to one location triggers an event), StoryTrek triggered events depending on how you moved through the space (i.e. going forwards, backwards, stopping, or turning).

Greenspan, a professor at the University of Carleton who spearheaded the project, says it's that spatial narrative that draws people into locative media and adds a level of immersion we don't get in other media.

"Every time we get a new test group in…people seem really thrilled and excited when they realize somehow this device knows where they are. It can respond to some extent to their surroundings," he explains. "In a way, it becomes an almost uncanny experience where they experience the world once in actuality through their senses and then re-experience the same world through the device and the story that the device is triggering."

At the same time, because of the unpredictable nature of navigating through physical spaces, these locative experiences bring out a level of serendipity that creators could not have anticipated.

This happened when Khaled and Barr were playtesting Crisis 22. In the story, there's a moment where a cyclist rides past the character. During testing, that very moment was triggered at the exact moment a real cyclist rode past them. As Barr explains, it almost felt like a crossover where the cyclist of the story emerged into the real world.

Instances like these make players wonder how much the game really knows about their environment (even though in the game's case, it was a mere coincidence). It's this layer of ambiguity—of not knowing what the game knows or doesn't know—that sparks the imagination of players.

Greenspan saw this possibility during disconnected stories, where players experienced the narrative out of order. Amazingly, they didn't find it confusing: "People insisted on finding meaning and patterns in what they've experienced…they would fill in the blanks and create a seamless narrative experience," he says. "So effectively, they re-wrote the story in a completely different way than we anticipated."

Despite its potential, locative media brings its own set of challenges. Khaled, StoryTrek's initial developer and now a professor at Concordia University, says creators need to consider how these spatial experiences impact those who aren't playing.

She points to an experience a male tester had while playing another story made on StoryTrek (this time around, it was about a zombie apocalypse). The story prompted him to find and join a nearby group of people for safety. The only issue? The nearest crowd was a group of children on a school trip. He ended up not doing it as it would have looked "super creepy" to find a stranger randomly joining a group of small children.

"[As a player], you can read other people into your experience because you're having this locative experience but other people don't necessarily know what you're doing," says Khaled. "There's a kind of imbalance there, you can even call that an ethical imbalance. [Others] become players in your game and your story but in a sense you're using their participation unknowingly."

Rueb sees a similar effect in the sense that oftentimes players get too immersed in the augmented "imaginary" space. It's not usually a bad thing, but it becomes a problem when people start running around in traffic, being unaware of their bodies in space, or their relationships to the larger social space they're part of.

While Pokemon Go may have brought the medium's largest audience to date, a lot of the people we talked to see it as quite primitive and simplistic. The game's biggest problem lies in its limited interest with the physical space it inhabits: Pokemon Go simply overlays a virtual world over the real one.

"The postmodernist in me is concerned that [this disregard] would lead to a flattening of reality rather than an enhancing of it," says Greenspan. "[To] truly augment reality, what you're looking for [are] those levers and hooks in the landscape that allow you to draw out some of the rich history, context, politics, and culture of the place that you happen to find yourself and narrativize that."

The beauty of locative media is this intersection between reality and the virtual. As with Khaled and Barr's own serendipitous experience with the cyclist, when these two worlds collide, it encourages players to look deeper at their physical environments to try to find meaning in things (no matter how mundane).

When the technology allows for ambiguity—and players are able and encouraged to fill in the blanks—reality can become hyperreal and players oscillate between the game world and real world to "complete" their experience.

One of the other ways artists have introduced this ambiguity is by removing the digital screen altogether. In location-based games like Blast Theory's A Machine to See​ With​ and Rueb's own works, the focus is creating an augmented aurality experience.

"Sound works on more of a subliminal level than visual stimuli," says Karen Collins, a University of Waterloo professor who specializes in interactive audio. If you look at good horror films, the scary parts usually aren't shown but are left to the audience's imagination.

"By using sound, it brings up a lot more memories than images…we don't pay as much attention to it as the images [so] it kind of works in that subconscious level where it's underneath the surface…it's so evocative and brings up so much of our own experiences, our fears, and so on."

As seen in the works of Greenspan, Khaled, and Barr, having this space where players can inject their own experiences makes those moments uniquely personal. At the same time, the combination of a purely sound-based experience mixed in with locative features forces players to fully focus on their surroundings.

"It's really easy in the abstract for people to think, 'Oh, it's a soundtrack like an audio tour,' and it's just not," says Rueb of her works. "When you have responsive media in a physical space and your body is activating it and it in turn is giving you…subtle indications of where to go, what you might want to explore, what might be around the corner, all of those are kind of layered spatial experiences. It's a social-spatial experience really. As a compositional space, there's just so much more to explore."

 A Machine to See With Trailer

One of these explorations is dealing with the blurring between reality and the virtual. Adams explored this theme with Blast Theory's 2010 game, A Machine to See With. In the game, players play the "lead" in a heist movie where they are preparing to rob a bank.

Instructions on the phone guide and prompt them into thinking about how they would do the robbery (i.e. where is the nearest police station, how would you escape, etc.). Despite the auditory nature of the game (or perhaps because of it), the line quickly blurs between pretend robbery and real robbery.

"What we immediately discovered is that of course we feel incredibly guilty doing this," says Adams. "It feels like you've already begun a criminal enterprise when all you're doing is talking on the phone with someone else."

Like other locative experiences, because players come into the game primed with their own set of experiences (in this case, that of a heist movie), physically going through the steps of an action (i.e. preparing for a robbery) starts to feel real. When "playing" within a physical space with a very real bank, that magic circle where players are cut off from reality disappear. Here, they're placed right in the thick of things.

While the game, of course, never goes through the robbery, letting players "fill in the blanks" (by prompting them to think about the robbery) puts them in a situation where they start feeling like it's the real deal. After all, from the outside, there is nothing different from what they are doing versus what real robbers would do. The only thing separating them is that phone signalling their true intentions (of "just" playing).

This is the kind of intersection that artists like Rueb sees locative media bringing up when we engage with physical spaces. While we understand our environments in both conscious ways (whether social, physical, etc.) and unconscious ways (where we're "bringing ourselves to a place, a moment or a time, and becoming part of it"), our gradual transformation into a simulation-driven society (or one tied to our screens) is making us lose that connection she finds integral for survival.

"We kind of lose touch with some of that intrinsic body knowledge that is both functional and also a source of sensory engagement…a kind of [engagement] with your physical environment as a whole complex set of inputs if you will," she explains. "It's a kind of sensory complexity we often take for granted. We kind of think of it as 'dumb space' when in fact it's one of the most vivid and complex environments."

Perhaps with locative media, we can re-discover what's been there all along.

Follow Nicole Pacampara on Twitter .