AI Directives Hidden in Court Submissions: A US First Leads to Sanction

Attempt to Sway the Outcome

According to НВ — Техно: Concealed AI instructions were hidden inside a complaint filed in the United States.

August 17, 12:30

A plaintiff representing himself took the unusual step of embedding hidden text prompts for AI into court documents, aiming to alter the outcome of litigation against New York Bariatric Group. The instructions were typed in white and therefore not visible on a standard review. The filing accused the medical group of invasion of privacy, discrimination, and additional grievances.

Those invisible messages contained language meant to make the AI generate answers consistent with the submitted document and to advocate for the plaintiff's success. Such manipulation of generative AI is widely called a prompt injection. It is important to note that the court does not use AI to process evidence; the hidden text was uncovered by a court clerk.

The Plaintiff's Next Moves

Even after being caught, the plaintiff inserted further hidden notes into later filings, among them a reference to an episode of the cartoon "SpongeBob SquarePants." Connecticut Superior Court Judge Walter Spader Jr. cautioned that pro se litigants do not have unlimited freedom. Consequently, the judge prohibited the plaintiff from using the electronic docket system, forcing him to submit all documents physically at the court clerk's office.

According to the plaintiff's statements to 404 Media, he was carrying out an audit of the judiciary to discover whether AI was being used. JD Supra reports that this is the first documented instance of prompt injection in a U.S. court case for which the responsible party was penalized.

The episode illustrates the complications and dangers that AI integration brings to legal proceedings. While the courts have yet to employ AI for examining submissions, this case calls into question the ethical and legal boundaries around technology in the justice system. The legal sector must respond to these new technological pressures to maintain fairness and openness in future cases.