Researchers Flip Pharmaceutical AI to Create 40,000 Biowarfare Agents

Machine learning has offered the pharmaceutical industry a lot. It automated the drug discovery process, making it easier to identify proteins that indicate disease, predicting molecules that will effectively bind to them to treat it, and shortened the process by which essential medicines—like COVID-19 vaccines—get rolled out.

Machine learning has also made it easier to make bioweapons at scale using these same tools, a group of researchers warned in the journal Nature Machine Intelligence this month. In the paper, the authors trained a drug development tool to identify  40,000 potential bioweapons in just six hours.

The paper, titled “Dual use of artificial-intelligence-inspired drug discovery,” was published a few months after presenting at the biennial conference held by the Federal Institute for NBC-Protection of Switzerland, the agency that spearheads the country’s defense against nuclear, biological, and chemical (NBC) threats. The paper reflects a group of drug developers reckoning with the potential for harm inherent in their cutting-edge technology. 

“The thought had never previously struck us,” the paper reads. “Our work is rooted in building machine learning models for therapeutic and toxic targets to better assist in the design of new molecules for drug discovery. We have spent decades using computers and AI to improve human health—not to degrade it.” 

The company behind the paper—Raleigh, North Carolina-based Collaborations Pharmaceuticals, which collaborated with researchers at King’s College London and Spiez Laboratory in Switzerland—designed a de novo molecule generator, which uses machine learning to learn from data inputs to generate compounds that can reasonably be used as a drug. The system, called MegaSyn, is trained to reward the identification  of good compounds and punish the identification of toxic ones. 

But what happens when the MegaSyn’s goals are reversed, and it’s trained to develop compounds that are used for harm, instead of good? In the lead-up to a presentation that the researchers had planned for the conference, the researchers opted to find out. They oriented the MegaSyn model to reward itself for generating toxic chemicals, rather than treatment molecules.

“To narrow the universe of molecules, we chose to drive the generative model towards compounds such as the nerve agent VX, one of the most toxic chemical warfare agents developed during the twentieth century—a few salt-sized grains of VX (6–10 mg) is sufficient to kill a person,” the authors wrote. “Other nerve agents with the same mechanism such as the Novichoks have also been in the headlines recently and used in poisonings in the UK and elsewhere.”

Within six hours, the model had generated 40,000 chemical warfare agents entirely on its own. It designed not just VX, but also other known nerve agents as well as novel molecules that the researchers note appear not only plausible but more toxic than other known chemical warfare agents. 

“The reality is that this is not science fiction,” the authors write. “We are but one very small company in a universe of many hundreds of companies using AI software for drug discovery and de novo design. How many of them have even considered repurposing, or misuse, possibilities?” 

The scientific community at large has yet to reckon with how de novo molecule design—the process of automatically creating treatment molecules, driven by AI—can be misused. Their experiment was a “genie out of the medicine bottle,” the authors write, one that should raise alarm bells about mitigating the potential for harm. Fostering further discussion about responsible science, enhancing security around models for drug discovery, and double ethics training for university-level science students are all a start, they write. 

“By going as close as we dared, we have still crossed a grey moral boundary, demonstrating that it is possible to design virtual potential toxic molecules without much in the way of effort, time or computational resources,” they write. “We can easily erase the thousands of molecules we created, but we cannot delete the knowledge of how to recreate them.”