This Small Company Is Turning Utah Into a Surveillance Panopticon

The state of Utah has given an artificial intelligence company real-time access to state traffic cameras, CCTV and “public safety” cameras, 911 emergency systems, location data for state-owned vehicles, and other sensitive data.

The company, called Banjo, says that it's combining this data with information collected from social media, satellites, and other apps, and claims its algorithms “detect anomalies” in the real world.

The lofty goal of Banjo’s system is to alert law enforcement of crimes as they happen. It claims it does this while somehow stripping all personal data from the system, allowing it to help cops without putting anyone's privacy at risk. As with other algorithmic crime systems, there is little public oversight or information about how, exactly, the system determines what is worth alerting cops to.

In its pitches to prospective clients, Banjo promises its technology, called "Live Time Intelligence," can identify, and potentially help police solve, an incredible variety of crimes in real-time. Banjo says its AI can help police solve child kidnapping cases “in seconds,” identify active shooter situations as they happen, or potentially send an alert when there's a traffic accident, airbag deployment, fire, or a car is driving the wrong way down the road. Banjo says it has "a solution for homelessness” and can help with the opioid epidemic by detecting “opioid events.” It offers “artificial intelligence processing” of state-owned audio sensors that “include but may not be limited to speech recognition and natural language processing” as well as automatic scene detection, object recognition, and vehicle detection on real-time video footage pulled in from Utah's cameras.

In July, Banjo signed a five-year, $20.7 million contract with Utah that gives the company unprecedented access to data the state collects. Banjo's pitch to state and local agencies is that the more data that's fed into it, the better its product will work. Thus, the company has spent the last year trying to get as many state and local agencies as possible to give it access to its CCTV and traffic cameras, audio sensors, and other data.

On this it has been incredibly successful. Banjo has installed its own servers in the headquarters of the Utah Department of Transportation (UDOT), a civilian agency, and has direct, real-time access to the thousands of traffic cameras the state operates. It has jacked into 911 systems of emergency operations centers all over the state, according to contracts, emails, and other government documents obtained by Motherboard using public record requests, as well as video and audio recordings of city council meetings around the state that we reviewed.

Its contract with the state says that Banjo’s technology will be deployed or is in the process of being deployed in all 29 of Utah’s counties, in the state's 13 largest cities, and in 10 other cities with “significant relevance” as well as for “campus security” for the University of Utah. A representative for Banjo told the city of Springville, Utah in January that a total of roughly 70 other cities and counties within Utah had agreed to give Banjo their data. It is also working with the Utah Department of Public Safety and the Utah Highway Patrol, according to public records.

So far, however, the Utah Attorney General's office could not provide Motherboard with an example of a real-life case in which Banjo has been used, though it insisted that it had indeed been used.

Banjo is yet another in a new crop of predictive policing and artificial intelligence-focused policing tools. Unlike other tools, such as Palantir—which is used by ICE and is so controversial that there have been multiple protests about its use—Banjo claims that it does not help police find criminals, it helps them to find "emergencies" and locate criminal acts.

“We essentially do most of what Palantir does, we just do it live," Bryan Smith, Banjo's top lobbyist, told the Salt Lake Valley Emergency Communications Center Operations Board, which is made up of police chiefs and 911 dispatch officials, in August. "So Palantir is a tool you use for analysis, kind of to deep dive investigate certain things. What we want to do is deliver you the information right at the moment. Think of a Google Maps [you’ll get] a pin that drops with a [computer-aided dispatch] caller and attached to that event, within sub one second with intelligence related to that event.”

At the meeting, Smith aimed to get Banjo access to 911 call information, which the company has since received, according to emails with local police. He said that "we strip out all personally identifiable information. We're not trying to ID a suspect at all. What we're trying to do is point you guys to a problem. We're telling you where the needle in a haystack is, not who that problem is."

What remains unclear is how a company that proposes to tell police of "anomalies" as they happen could protect people's privacy, or operate without serious biases, two largely intractable problems with surveillance and AI.

Banjo also proposes to monitor thousands of public surveillance cameras in real time with an algorithm rather than a human, something that has terrified security and civil liberties experts for years.

"We don't have enough eyes to watch all the cameras, we don't have enough people to understand all of the different signals, so what we use the system to understand those signals," Smith said at the meeting.

Privacy experts are unsure how Banjo can be doing anything other than applying machine learning to a terrifying amount of data to create a persistent panopticon pointed at everyone who lives in Utah.

"I've run out of adjectives to describe how upsetting and dystopian things like this are," Chris Gilliard, a professor at Macomb Community College who studies police surveillance and privacy, told Motherboard after reviewing parts of Banjo's contracts with the state.

"This rise of video analytics, of being able to not just monitor but analyze everything that’s going on in overwhelming streams of data is a game-changing power for police," Andrew Ferguson, author of The Rise of Big Data Policing: Surveillance, Race, and the Future of Law Enforcement, told Motherboard. "The ability to see and categorize and understand everything is something we’ve never had before. AI-infused eyesight is a superpower that we haven’t seen before in policing, and is something we really should have rules about and discuss whether that’s OK."

Banjo originally had no connection to Utah, but its business is now intertwined with the state. It moved its headquarters from Las Vegas to Park City in 2018. Smith, its chief lobbyist, is the former head of Utah's Republican party and a former Chief of Staff in Utah's State Senate. Both the Utah Office of Economic Development and the Economic Development Corporation of Utah have published interviews with Patton that have largely focused on how Utah is a nice place to run a business: “I can’t imagine any better place than Utah,” Patton said in one of the interviews.

In 2008, at the age of 35, Patton was clean shaven and had just founded a FindAnyFloor.com, a flooring "information site and social community." Patton's bearded, trail-blazing persona has become a core part of Banjo's story.

“He looks like a guy you think would be playing a banjo,” Smith told the city council of Lehi, Utah, in December while pitching the company at a council meeting. “He’s not your typical tech CEO."

Patton has told Banjo's origin story many times. As his story goes, he tweeted about waiting for a flight back to his home in Las Vegas while at Boston Logan International Airport. When he got home, he saw that his friend, who served with him in Desert Storm and who he hasn't seen in 15 years, posted on Facebook about being at Logan at the same time. Both friends were at the same place, at the same time, and said so on social media, but still managed to miss each other like ships in the night.

Banjo was originally designed to fix this missed opportunity. By scanning public social media feeds and comparing them with your interests, the app's aim was to let users know when their friends are nearby, if their favorite band happens to be playing in the city they're traveling to, and so on.

The company trudged along like this for a few years—a location-based social media aggregator for consumers. According to Patton, the company's plans changed entirely on April 15, 2013, the day of the Boston Marathon bombing.

"The moment that bombing happened, none of us knew it was a bombing. We knew it was an explosion. I went immediately into our system and said, 'Can I rewind time and go back to that location where that explosion was and know what happened before the explosion occurred?'" Patton said.

"In that moment in time I discovered literally what had happened before the explosion went off, who the people were around that, when the explosion went off, what was happening, and that it was a bomb. I went to the [company] board that day and said 'the company is changing.' If we could automate—if we could build AI that could have known this ahead of time, and certainly know this in real-time, imagine how that could have changed everything. At that moment our mission changed and from that moment on we've been building the crystal ball."

(Peter Thiel's surveillance and big data analytics company Palantir, incidentally, is named after a crystal ball from Lord of the Rings, which allows the person who wields it to see any part of the world, past or future.)

Plenty of companies, such as Geofeedia, have sold tools to law enforcement that scrape social media to alert or inform police on what is happening in a particular area. But Banjo differs in that it not only uses social media posts, but also has direct access to state data, such as camera feeds.

"Twitter shut down Geofeedia’s access and they shuttered themselves. Even public-focused social media with geolocational technologies could be invasive to associational freedoms, privacy and the rest. This seems to be that technology on steroids," Ferguson said.

Banjo did not respond to four emailed requests for an interview sent to Patton, Smith, another executive, and its general inbox. They also did not respond to a detailed list of questions for this article, which asked for more specifics about its anonymization process and patents, statistics about Banjo's use in Utah, any examples of when it had been used in a real-life situation, its data access and protection protocols, and any work it has done outside Utah.

"Our feeling is the best way to predict the future is to invent it," Ric Cantrell, chief of staff for Utah's Attorney General, said in an interview. "What if we could invent it in a Utah-specific way that, on one hand, understands that need for law enforcement to have powerful tools to do their job, but on the other hand respects the right of every American citizen to believe what they want, to say what they want, to associate how they want, to be free from government intrusion?"

Emails obtained via public records request between Banjo and Utah officials show how the company pitches its services to agencies across the state. In one instance, Patton himself tries to ingratiate himself to police chief Ken Wallenstine around a hobby they share: riding motorcycles. In an email to the chief of the West Jordan, Utah police department, Patton wrote: “Happy December - which only means you are somewhere warm riding right now. I wish I was there! For [Salt Lake County], there are only a couple of people that aren’t on our radar yet. One is the chief of South Salt Lake. I heard he likes to ride, so I am going out on a limb here and seeing if you know him well. I would like to get our team over there and showing him Banjo.”

In January, Wallentine emailed Patton to ask if he was "at all interested in naked land for building" in the city. "I chatted to our economic development guy and he might have some space ideas. We'd love to have Banjo in West Jordan!" A public information officer for the city of West Jordan told Motherboard that it is "aware" of the company but that it is "not utilizing Banjo as part of its public safety operations," though the city's police department and mayor signed an agreement giving the company access to its police dispatch system, public safety cameras, and other city data in January. On January 30, Smith emailed officials in the city to "coordinate on implementation of your signals and starting to schedule trainings for your officers."

In another email to Gary Giles, chief of police in Orem, Utah and president of the Utah Chiefs of Police Association, Smith appears to offer him help while pitching him on Banjo's services.

"I just wanted to check in with you here at the start of the year on getting together about getting Banjo for you and your team," Smith wrote in a January 9 email to Giles. "Do you and/or your command staff have any times in the near future we could get together?"

"I am limited on when I can do it as the legislative session is coming up and I am going to be in Salt Lake quite a bit," Giles wrote back.

"I totally understand about the legislative session - Congrats by the way," Smith replied. "I was the Deputy Chief of Staff in the Utah Senate before jumping over here to Banjo. I'd also love to chat about how I could support what you are doing up on this hill this session."

Giles told Motherboard that he did not talk to Banjo about any legislation. "My conversations with Banjo have never been about anything except them wanting to give us a presentation," he said. "Other than meeting with Bryan at Orem PD (I believe in January), I have not spoken with him."

While Banjo is based in Utah and its website lists three offices in the state, the company also has offices in Redwood City, California; Washington, D.C.; and Las Vegas. The company's largest and most advanced presence appears to be in Utah, though Banjo has mentioned in presentations to police that it also works in Arizona. According to Crunchbase, it has raised over $120 million from investors, most of which came from a $100 million Series C round of investments led by Softbank, which famously invests in WeWork, Uber, and a host of other startups from its $100 billion "Vision Fund." Banjo now claims to hold one of the largest "live data stores" in the world.

Patton talks regularly about ethics in AI. While speaking to students at Stanford in January 2019, Patton said that unlike his contemporary CEOs in Silicon Valley who collect and monetize private data with advertising, he believes tech can be "altruistic." He explains that Banjo collects data in order to save lives, and that it's able to do so while keeping people anonymous and protecting privacy.

"From day one we've stripped personal identifiable information (PII) from all of our data, where even our engineers cannot deanonymize it," Patton said at a conference last year. Its stated mission is to "reduce human suffering."

Banjo's pitch to Utah from the beginning has been that it finds crime without identifying criminals. It looks for cars without looking for who is inside the cars. It detects riots and protests without telling cops who is at them. It detects drug use hotspots without identifying the drug user. Patton often mentions his company has patented technology to strip PII from publicly available data, but patents Motherboard found do not go into detail about how that is done.

Ferguson said it's at least laudable that the company is even mentioning privacy: "I do think a company should get some credit for trying to take some steps to recognize the risks of personal identifiable information in their system and trying to limit the privacy-invading nature of the privacy-invading surveillance technology it's building."

Cantrell, of the Utah Attorney General's office, said that Banjo is one of the Utah startup scene's crown jewels, and that its policy on privacy is what convinced him to set up a meeting.

"One of the founders of the Silicon Slope movement called our office and said, 'Hey, I just met a dude, and what he is engaged in is pretty phenomenal. You need to come meet him.' And so we did," Cantrell said. "it became pretty apparent that this emerging technology could help us in our fundamental constitutional and statutory responsibilities to keep the people of Utah safe. We respected the company because of their careful attention to keep private data private."

In 2018, Cantrell's office set up a child abduction drill with law enforcement in the state and invited Banjo to participate. Smith later described the drill to the city council of Lehi, Utah, in a December 2019 presentation. In that drill, police officers—a "staff of 100," Smith said—took eight hours to recover the simulated child. "A stranger abduction at eight hours, you're looking for a body at that point more than a person," Smith said. Banjo, he said, located the child in 27 seconds "and we could have helped them recover the child in four minutes."

Smith didn't explain how Banjo was able to locate the simulated child so quickly, but during his presentation he shows a powerpoint slide that has dozens of data points feeding into its system: "social media, 911 calls, weather, utilities, air quality, public cameras, traffic cameras, emergency services, autonomous vehicles, flight info, public cameras." In its contract with the state, it also says it can do "data signal ingestion for IoT Sensors," but does not elaborate. Banjo also claims to have a partnership with Waze, the Google-owned navigation app which tracks traffic with phone location data. (Waze did not respond to a request for comment.)

In his demo, the only data that has any names attached, Smith says, is the child sex offender registry, "because that's public information." Smith's presentation to Lehi was intended to (and did) convince the city council to share access to its own public safety and CCTV cameras, automatic vehicle location for city-owned cars, its record management system (with PII removed), and its computer-aided dispatch system with the company, which would in turn make both the city and the entire state safer. The city council adopted a resolution to work with Banjo a week after his presentation, and the city officially signed its contract at the end of January, according to an executed contract obtained using a public records request.

In return for cities' data, Banjo says it will send alerts to cops when there's an "emergency," giving them the location and time of the incident as well as information about that incident. Police officers who are trained to use Banjo can also log into a web portal and look at a specific area to see what types of signals Banjo is collecting. They can also look at a map and "rewind time" to look at the locations of public transportation, CCTV cameras, and look at other data Banjo ingested at that time.

The closest Banjo gets to describing exactly how its technology works and what it will do with that data is related to its supposed processing of the 2017 Las Vegas shooting at Mandalay Bay.

“Literally before law enforcement was on the scene, we were able to detect there was a mass incident going on and where it was at, based on a variety of signals in the area,” Smith told the Lehi city council.

He said the company had parsed a Facebook Live video in which "a gentlemen in Spanish said something about gunshots. We don't know who that person was." Meanwhile, a mile from where that Facebook Live video was being shared, camera footage Banjo ingested suggested there was a “riot” occurring on the Las Vegas Strip. “But it wasn’t a riot, it was just people running in patterns across the road in a way they don’t normally do. That, in combination with a tweet, we were able with confidence to say a mass shooting was happening. Sadly, we weren’t working with law enforcement at the time.” Las Vegas Metropolitan Police said in response to a public records request that it has never had a formal conversation with Banjo.

In his demo to Lehi, Smith explained how the company now has its servers within the Utah Department of Transportation's headquarters, and that it is now collecting and analyzing every traffic camera it operates: "Their cameras have never been able to be used for law enforcement or public safety despite having over a thousand of them," he said.

Smith explained that Banjo's AI can look for specific types of cars on UDOT camera footage: "We're helping law enforcement find the child in this event, not the perpetrator. We're not helping to lock them up or criminalizing anyone. We're not. […] We're not looking for people or gender. It's cars. They're UDOT cameras, they're transportation cameras, they're made for cars, so we're going to look for cars."

"Again, this is specific to a child abduction," he added.

Emails and contracts obtained by Motherboard from the Utah Department of Transportation using a public records request show that Banjo's initial contract with UDOT allowed for the platform's use only in "child abduction cases." This contract was amended in December to allow for all "emergency event detection," according to emails between the Utah Attorney General's office and the Department of Transportation, this request came at the request of the Utah Department of Public Safety. The Utah DPS said in an email that it currently "uses Banjo for traffic monitoring and response and not for any criminal investigations."

The emails also confirm that Banjo has installed its own server infrastructure behind UDOT's firewall. Before Banjo, the UDOT never saved video from its cameras and instead relied on them to give real-time traffic updates. Banjo is now recording the footage and making it available for 24 hours, after which it will be deleted, according to the contract, emails, and presentations Smith has given.

Cantrell, of the Utah Attorney General's Office, said that his office and state police are primarily focused on the possible use of Banjo in a child abduction case, but noted that Banjo has never been used for that purpose and that there has not been a "major kidnapping event" in Utah in the last two years.

"If a kidnapping occurs in certain areas where Banjo is now covering, we have hope that we could bring a child home safe where we wouldn't have been able to in the past," Cantrell said. "If a child gets kidnapped, and I'm painting with a broad brush, but generally, statistically, it’s going to be an 11-year-old girl, it’s going to be … I don't want to get too gruesome."

There is a long history of surveillance products being sold to the public using worst-case scenarios—child kidnappings, terrorism, mass shootings—and then quickly trickling down to be used for more mundane purposes. Previously sold to mostly law enforcement agencies, schools now use facial recognition technology. IMSI-catchers, which track cell phones' locations, are now used to investigate a wide range of crimes. iPhone cracking technology, once the center of a federal standoff between the FBI and Apple in the aftermath of the San Bernardino mass shooting, is now regularly sold to state and local cops.

The cases in which Banjo could be used has already been expanded. After initially being interested in child abductions, Cantrell's office has also signed up for Banjo's "opioid module," which the Utah Department of Health told the Utah state legislature was developed for $250,000 as part of a U.S. Drug Enforcement Administration program.

Banjo's opioid module, according to the Utah Department of Health, is designed to be a "live heat map and trend analysis of opioid-related events." The module will "provide agencies, such as the DEA, Attorney General's Office, and law enforcement entities within Salt Lake County with live insight into the crisis so proactive and immediate support resources can be effectively deployed." Public documents reviewed by Motherboard don't explain what data Banjo uses for this system, but Cantrell said that it will take into account "emergency room visits, morgue visits, naloxone use, which beds are open and where for addiction recovery."

Dr. Daniel Ciccarone, a professor at the University of California, San Francisco, who studies the heroin and opioid epidemic, told Motherboard he is skeptical of this plan because "they're not going to have very good data," and because the organizations that will actually receive the information are law enforcement, not harm reduction specialists. "What's the purpose of the AI? How will it help? Is it to make the criminal justice system harsher? We're starting to see things like that elsewhere, and it's UNPRODUCTIVE, capital letters, underscore, bold font," he said. "I would be very worried about any technology set to give information to law enforcement for the purposes of public safety."

Anne Boustead, an assistant professor at the University of Arizona who studies electronic surveillance and drug policy, said the data is likely to be skewed: "I have concerns about whether a tool such as this would provide an unbiased picture of opioid use in Utah. If they are drawing on data is disproportionately gathered from certain places/people, then they are going to be more likely to notice when an opioid event (and to be honest, I’m not really sure what that means?) occurs in those places," she said.

"If they are drawing inferences based on information collected from, say, security cameras, then they are going to be more likely to notice events that happen where security cameras have been placed," she added. "If these are located in socioeconomically disadvantaged communities, then it could lead to inferences that opioid events are more likely to occur in these communities—when, in fact, it is just more likely that this will be observed."

There is, again, almost no information about how Banjo actually works, what its algorithm considers to be an "anomaly" or an "emergency event," and how it will be deployed.

"Do people implicated under [Banjo's programs] have any means of examining it or calling it into question?" Gilliard said. "Will people who might be potentially exonerated by a system be able to request information and documents? Usually the answers to these questions are ‘no.’"

Banjo will, again, direct police to the anomalies and emergency events it detects in real time. Even if all names are stripped from the data, there will be real people at the other end. Maybe that person will be a kidnapper or mass shooter. Maybe it will be a truant student, or a drug user, or a protester. These people will, inevitably, be forced into potentially dangerous interactions with the police.

"It's only a small step from going to predicting something that might happen, and figuring out a way to cause something to happen," Dave Maass, a senior investigative researcher at the Electronic Frontier Foundation, told Motherboard. "I do not believe anything that a company offering predictive systems tells me anymore … I don't even know how you'd show efficacy."

Patton, for his part, says that there is only one statistic that matters to the company:

"We measure only one KPI [key performance indicator] at Banjo. Truly one KPI across the board, and that's lives saved," he said.

What is unclear is how many lives Banjo has saved, if it has saved any, and whether it's even possible to create a product that automatically detects crime without creating a creeping surveillance state that inherently infringes on people's rights and invades their privacy.

While it's still very early days for its implementation in Utah, we have no idea whether it has been useful in the real world. Cantrell couldn't identify a single case that Banjo's technology had helped on. A public records request sent to Utah's Highway Patrol requesting any case reports in which Banjo was used returned no documents. Police departments who have signed up for Banjo told Motherboard that they have not actually used it. This doesn't mean, of course, that Banjo has never been used or that it's not been helpful, it's just that neither the company nor the state of Utah has been able to produce any evidence that it has.

"They do have case studies," Cantrell said. "I'm waiting for case studies from Banjo. I'm still waiting for information from them."

Cantrell suggested there is a bright future ahead for Banjo. The state is exploring a "homelessness module," and that if it wanted to, it could add a "meth module" or a "human trafficking" module. A homelessness module would "chart beds open at homeless shelters, spikes in crimes, 911 calls, things like that." Cantrell said you can never really know what the technology might be useful for. Police officers in West Jordan have openly mused in emails about getting a government grant to deploy Banjo in schools.

"In the next 50 years there’s going to be incredible emerging technologies that law enforcement, governments, citizens have never thought of before, and we want to make sure that we utilize appropriate technology to keep people safe," Cantrell said. "Which is our mission, right?"