US Supreme Court Warns Against AI Use Within the Legal System

Top concerns include false claims driven by AI hallucinations, biased data sets, and potential violations of confidentiality.

Artificial intelligence has gotten its day in court: A new report from the US Supreme Court is cautioning lawyers and clients about the dangers of using AI to help them with their cases.

Many AI tools are free for anyone to use — from ChatGPT to Bard — and they’ve been doing just that. Now, the word about certain lawyers who have been caught submitting briefs with fake ChatGPT-created citations seems to have trickled up to the highest court in the land. They have some thoughts.

The new report, titled “2023 Year-End Report on the Federal Judiciary,” is written by US Chief Justice John G. Roberts, Jr., and focused entirely on AI. Here are the big takeaways.

Downsides of AI for Court Cases

One of the biggest issues with using AI within the legal system: AI tends to generate “hallucinations,” a result of the fact that AI is designed to produce concepts and sentences that feel real, rather than ones that actually are real.

As Roberts dryly puts in the report, lawyers who have used generative AI bots end up submitting briefs “with citations to non-existent cases (always a bad idea.)”

Surfshark logo🔎 Want to browse the web privately? 🌎 Or appear as if you're in another country?
Get a huge 86% off Surfshark with this special offer.See deal button

Other AI red flags include the legal concerns surrounding the entering of confidential information into AI tools at all, plus the likelihood of biases that are baked into the data that AI is trained on and which are then replicated in the AI’s output.

“In criminal cases, the use of AI in assessing flight risk, recidivism, and other largely discretionary decisions that involve predictions has generated concerns about due process, reliability, and potential bias.” – US Chief Justice John G. Roberts, Jr.

Added together, the challenges of hidden bias, outright lies, and legal red tape make AI sound like a pretty terrible fit for the legal system.

AI Does Have Benefits in a Legal Context

The Supreme Court’s report isn’t a complete polemic against artificial intelligence. It notes that targeted AI use can be a helpful time-saver, giving some limited tools to those who can’t afford a human lawyer.

“Proponents of AI tout its potential to increase access to justice, particularly for litigants with limited resources. […] For those who cannot afford a lawyer, AI can help. It drives new, highly accessible tools that provide answers to basic questions, including where to find templates and court forms, how to fill them out, and where to bring them for presentation to the judge—all without leaving home.” – US Chief Justice John G. Roberts, Jr.

Used as tools to open up basic processes to a wider audience, AI can help make up for some of the imbalances in our current court system.

However, this relatively quick disclaimer is about all that Roberts has to say about the positive benefits of AI as it currently exists. In the very next paragraph, he notes the “caution and humility” required when using AI.

AI’s Full Impact Is Yet to Come

Ultimately, this new report isn’t saying anything new. We know that AI tools have a place, but that a human eye is key to ensuring that the final result is actually accurate.

However, the fact that the Supreme Court is taking the time to address AI’s place in the legal system is yet another sign that this relatively new technology is already reshaping a massive number of industries.

The final outcome of AI’s impact on every area of life has yet to be seen. But, if the Supreme Court has anything to say about it, that impact will be closely monitored.

Did you find this article helpful? Click on one of the following buttons
We're so happy you liked! Get more delivered to your inbox just like it.

We're sorry this article didn't help you today – we welcome feedback, so if there's any way you feel we could improve our content, please email us at

Written by:
Adam is a writer at and has worked as a tech writer, blogger and copy editor for more than a decade. He was a Forbes Contributor on the publishing industry, for which he was named a Digital Book World 2018 award finalist. His work has appeared in publications including Popular Mechanics and IDG Connect, and his art history book on 1970s sci-fi, 'Worlds Beyond Time,' is out from Abrams Books in July 2023. In the meantime, he's hunting down the latest news on VPNs, POS systems, and the future of tech.
Explore More See all news
Back to top
close Building a Website? We've tested and rated Wix as the best website builder you can choose – try it yourself for free Try Wix today