Lawyers Used AI to Make a Legal Brief—and Got Everything Wrong

By now, some AI-generated nonsense sneaking into legal briefs isn’t shocking. But that doesn’t mean the fallout is or should be any less severe.

In a case out of California, reported by The Verge, U.S. Magistrate Judge Michael Wilner slammed two law firms for filing legal documents riddled with fictional cases and quotes, all dreamed up by generative AI. By trying to cut corners, the two firms will now have to pay $31,000 in sanctions.

The offending brief was part of a civil lawsuit against State Farm. A plaintiff’s attorney used Google Gemini to whip up a rough outline. As AI often does, the outline Gemini spit out was filled with bogus citations. The document was passed on to representatives from the law firm of K&L Gates, where not a single person double-checked any of it before filing.

California Judge Calls Out Lawyers for Submitting Fictional AI-Generated Legal Briefs

According to Judge Wilner, “No attorney or staff member at either firm apparently cite-checked or otherwise reviewed that research.” Wilner only realized something was off when he tried to verify the citations himself and discovered, in his own words, “they didn’t exist.”

Not just flawed or outdated. Flat-out imaginary. Lawyering with as much expertise in the craft as an actor playing a lawyer on a TV show. He was almost duped into incorporating those hallucinations into a real court

Wilner issued an Order to Show Cause, and under oath, the lawyers confessed to using a mix of Google Gemini and Westlaw’s CoCounsel, and an AI legal research tool, to generate the initial draft. There was no disclosure, no checking, just blind trust in AI

Letting generative AI ghostwrite your legal briefs without vetting is still a career hazard, it seems. It was only a matter of time before the proliferation of AI chatbots made its way into the courtroom.

We’ve covered tons of stories just in the past few months chronicling the intersection between artificial intelligence and the US legal system, and they rarely have happy endings. Just ask the guy who used an AI avatar to argue in court for him.