Facial Recognition Company Lied to School District About its Racist Tech

Ever since they learned that Lockport City School District intended to install a network of facial recognition cameras in its buildings, parents in the upstate New York community—particularly families of color—have worried that the new system will lead to tragic and potentially fatal interactions between their children and police.

Now, documents newly obtained by Motherboard accentuate those fears. They show that SN Technologies, the Canadian company contracted to install Lockport’s facial recognition system, misled the district about the accuracy of the algorithm it uses and downplayed how often it misidentifies Black faces. The records, comprising hundreds of pages of emails between the district and the company, also detail numerous technical issues with SN Technologies’ AEGIS face and weapons detection system, including its propensity for misidentifying objects like broom handles as guns. 

Those false alarms are particularly concerning to parents. According to local media reports and documents obtained by Motherboard, when it was initially piloted, the system had automatic alerts when it detected weapons or certain people on the district’s “hot list.” The documents show its alert system evolved over time, and SN Technologies says there is a human in the loop at all times: "Our system has been designed and developed with safeguards to protect any false detection whereby a human must verify each individual alert before any alert is confirmed," the company told Motherboard after publication of this article.

“The police have said if they get a notification they’re going to treat it as a live shooter system, and you have a system that’s predisposed to make mistakes and misidentify people,” Jim Shultz, a Lockport parent, told Motherboard. “The risk of an accident, the risk of something horrible happening because the system is structured the way it is, to me, is 1 million times higher than [the chance] that the cameras are going to prevent a real situation.”

Lockport schools are 11 percent Black, and Renee Cheatham, a Lockport school board member and parent, told Motherboard that the thought of biased facial recognition triggering an armed police response has outraged Black parents and led to a “very, very dark time.” Both Shultz and Cheatham are plaintiffs in a lawsuit against the New York State Education Department (NYSED) seeking to ban the use of facial recognition in schools.

The Lockport school district and local police department did not respond to repeated requests for comment. SN Technologies did not respond to repeated requests for comment before publication, but sent a letter to Motherboard after publication with further information about its technology.

Lockport turned on the AEGIS facial recognition system in January, after several years of battling with parents, civil liberties groups, and the NYSED, which initially opposed the system before abruptly signing off on it in November 2019. At the time, Lockport was the first district in the state seeking to use facial recognition technology in its buildings.

In the months immediately before NYSED’s decision, Lockport officials received a series of warnings that SN Technologies was misrepresenting a crucial aspect of the AEGIS system.

In August 2019, KC Flynn, the company’s CEO, sent district officials a lengthy email discussing AEGIS’s overall accuracy and its rate of racial bias. He claimed that SN Technologies was licensing its algorithm from the French firm id3 Technologies, and that this particular algorithm had ranked 49th out of 139 in racial bias tests performed by the National Institute of Standards and Technology (NIST). The NIST tests demonstrated that all facial recognition algorithms perform worse on people of color.

But according to Patrick Grother, the NIST scientist who oversaw the testing, the agency never tested an id3 Technologies algorithm that matches the description Flynn gave Lockport officials. “Those numbers don’t tally with our numbers. They’re not even close. What id3 sent to NIST is not what these people are talking about,” Grother told Motherboard.

Furthermore, he described Flynn’s statements about how facial recognition accuracy is calculated as “completely nonsensical.” And Grother said he warned a lawyer representing Lockport about all those issues when they first shared Flynn’s email with him in 2019.

The district received even more evidence that SN Technologies was lying in October, when it received a report from Freed Maxick, the accountancy it had hired to audit SN Technologies’ claims. The auditors wrote that the company had inaccurately claimed that, in NIST testing, its algorithm misidentified Black men twice as often as white men and misidentified Black women 10 times more often than white men. In reality, the auditors wrote, NIST tests of the id3 Technologies algorithm that Flynn’s company claimed to be using actually showed that it misidentified Black men four times more often and Black women 16 times more often than white men.

Motherboard obtained a copy of the Freed Maxick report from the New York Civil Liberties Union (NYCLU), which is representing the Lockport parents in their lawsuit.

SN Technologies claims that it never misled Lockport: "Our relationship with the Lockport City School District over the last 4 years has been based on trust and transparency and all of our correspondence and communications is on the public record. The selection and adoption of the AEGIS system has been the subject of public meetings, discussions with various groups and published in local media outlets over the last 4 years," KC Flynn, president and CEO of the company, wrote in a letter to Motherboard. "I would like to categorically state, that SN Technologies Corp has never lied or misled the Lockport City School District including how our technology functions or the expected outcomes it provides."

Flynn also said " It is not uncommon for Facial Recognition algorithms to have differences in accuracy based on race and gender. Our AEGIS system performs at an accuracy level greater than 99.9% regardless of race or gender of an individual, as verified by the independent report conducted by Freed Maxick, one of the Top 100 CPA firms in the United States."

Flynn said that the company paid for the Freed Maxick review but was not involved in it: "SN Technologies is an expert, software development company. When the Lockport City School District asked us to provide feedback on the NIST report and our software, we provided our interpretation of the reported information and suggested that a third-party expert review and assess all the data and provide an independent report to the school district. As is the standard for these types of reviews, SN Technologies paid for the review but had no involvement in the review or the final assessment."It is unclear whether the AEGIS system has ever prevented an incident in Lockport’s schools, or whether police have ever responded to the schools as a result of a false alert. The district and the police department did not respond to requests for that data.

The emails Motherboard obtained through Freedom of Information Law requests show that in the short time the weapons detection aspect of AEGIS has been active, it has regularly misidentified objects as guns and has had to be recalibrated several times.

In one email sent to Lockport school administrators earlier this year, an SN Technologies representative says the company is training the system to recognize broom handles in an attempt to stop it from mistaking them for weapons. The following month, the system was still returning a series of false gun alerts.

Flynn said the system has been repeatedly tweaked since it was originally piloted: "Adjustments also had to be made after construction and renovations at the Lockport High School early in 2020, as the construction disturbed cameras and their connections to the AEGIS system. Once the install and settings were complete the system was fully functioning. It is our understanding from feedback we have received from the school district, that the gun detection tool is living up to expectations."

Shultz, one of the Lockport parents, said the sad irony of the situation is that as a result of the COVID-19 pandemic, facial recognition has been rendered useless because everyone entering schools must wear a mask. Meanwhile, other districts in the state used their Smart Schools money to pay for upgrades like new laptops and improved internet connectivity that are now even more vital to students’ success.

Lockport’s pursuit of facial recognition drew national attention to the district and prompted the New York legislature to pass a bill that would ban the use of facial recognition in schools until a comprehensive study of its effect on students can be completed. New York Governor Andrew Cuomo has yet to sign that bill, but state Sen. Brian Kavanagh told Motherboard that he has spoken to the governor’s office and expects it to become law this year.

But despite the legislation, the lawsuits, the media coverage, and local backlash in Lockport, other districts in New York have followed suit and are seeking state money for facial recognition systems of their own.

At least 11 other districts in the state have since applied for Smart Schools money to purchase facial recognition systems, according to a NYCLU analysis of the applications. Schools in other states, such as South Carolina, have also deployed similar systems which claim the ability to detect weapons and stop school shootings.

“There are other districts that are trying to get their hands on similar technology without the safeguards that should be in place,” Stefanie Coyle, the NYCLU lawyer leading the Lockport lawsuit, told Motherboard. “These tools are not the right way to go … [but] We now have companies that are targeting districts across the state.”

This article has been updated with comment from SN Technologies.