AI lawyer: what actually works?
An AI lawyer is useful. In the 2025 Vals AI test on US law, the best legal AI tools answered questions about a document more accurately than lawyers, scoring 94.8 out of 100 against 70.1. But it can invent citations to laws and court rulings, and the party to the case, in whose name the document is filed, is the one held responsible. Open every citation in the original source: a law on pravo.gov.ru, a court ruling by its case number.
Roman Kaufman, founder of Kaufman Lab · Data as of: October 2026

In May 2026, Rossiyskaya Gazeta (the Russian government's official newspaper) told the story of lawyers at one company who filed a cassation appeal, that is, an appeal to a higher court. It cited court rulings with direct quotes. Those rulings did not exist. An AI made up the cases and the quotes, and the lawyers did not check them. The court fined the company.
Which AI they used is not known. Would a paid legal AI tool have saved them?
AI is more accurate than lawyers on a document, but worse at editing contracts
In the Vals AI test, legal AI tools answered questions about a document more accurately than lawyers, but edited contracts worse. The Vals AI report on legal AI came out in February 2025. Four tools that volunteered for the test and a group of lawyers did the same law firm tasks on US law, and the lawyers did not know they were part of the test.
The table shows scores out of 100 on four tasks from the February 2025 report. Only one tool took part in multi-step research.
| Task | Best AI | Lawyers |
|---|---|---|
| Answers about a document | 94.8 | 70.1 |
| Document summary | 77.2 | 50.3 |
| Contract editing | 65.0 | 79.7 |
| Multi-step research | 55.2 | 70.1 |
AI leads where the answer sits in the document, while lawyers are stronger at editing and research. The authors name incompleteness as one of the typical mistakes of the tools. If you need to find several clauses in a contract, the AI often stops at one.
So you can go through your own contract with AI. But after AI edits a contract, read the edits yourself, line by line.
AI makes up citations, and the party to the case pays
A general-purpose AI does not answer from a checked database of laws. As a deputy CEO of a company that sells legal AI tools told Rossiyskaya Gazeta, such an AI produces "a statistically likely continuation of the text." That is why a court ruling it invents looks real, with a number and a quote.
That is what happened with that appeal. The Commercial Court of the West Siberian District fined the company 50,000 rubles (about $589 at the Bank of Russia rate of 84.90 rubles per dollar on October 11, 2026) for contempt of court (Article 119 of the Russian Commercial Procedure Code). Rossiyskaya Gazeta calls it the first such case in Russia. Under the court's ruling, the person taking part in the case is responsible for the accuracy of AI-generated text (Article 9, Part 2 of the Commercial Procedure Code). Lawyers wrote the appeal, but the party to the case here was the company.
If you file a claim in your own name, you are the party to the case. Following the logic of this ruling, the court will hold you responsible for the citations, no matter who wrote the text.
Courts ask the same of themselves. Under a concept paper of the Russian Supreme Court (order No. 59-P of September 3, 2026, as summarized by PPT.ru, a Russian legal and business news site), a judge must personally check every citation to laws and court rulings that an AI provides. Courts are bringing in AI step by step, with goals up to 2030. This is an internal document of the courts. For the parties to a case it is a guide, not a law.
In a 2024 test, paid legal AI was wrong in at least one answer in six
A paid tool does not protect you from invented citations. In 2024, researchers at Stanford RegLab and HAI asked legal AI tools more than 200 questions on US law. According to the Stanford test results, Lexis+ AI and Ask Practical Law AI gave wrong information in more than 17% of cases. Westlaw AI-Assisted Research made things up even more often, in more than 34%.
The products have been updated since then, but a subscription does not check a citation by itself. You check the answer of a paid AI the same way as the answer of a free one.
In our competitor analysis, a task that is not a legal one, checking with a word-for-word quote overturned 9 of the model's 29 "yes" answers. That is why we accept a "yes" from AI only with a quote that we can open.
Give AI the draft, but open every citation yourself
The company in that story would have been saved by one check before filing, because it would not have found the rulings it cited. You can trust AI with your document, but not with its citations. This is work where a person checks the result, like most tasks in our breakdown of what work to give AI employees.
- Ask the AI about your own document and ask it to list all the clauses on your question. Remove names and identifying details, and do not put client secrets into a public chat. Check what you can upload against the three groups in our guide on what data you can put into an AI.
- Open every article of law from the answer on pravo.gov.ru (Russia's official legal information portal) using "Расширенный поиск" (Advanced search), and compare the text before you sign. Find a court ruling by its case number on the court's website, in its case database.
- If the text matches, keep the citation. If not, remove it from the document. If the document is going to court and you are not a lawyer, show it to a lawyer with all its citations.
Frequently asked questions
What is an AI lawyer, and how is it different from a regular chat?
An AI lawyer is an AI built for legal tasks. It makes mistakes too. In the Stanford test, legal tools made mistakes less often than regular chatbots, which got 58–82% of legal questions wrong in an earlier test by the same authors. But in the Vals AI test in October 2025 on US law, accuracy came out the same: 78–81% for legal AI and 80% for ChatGPT. On citing sources, the legal tools scored 6 points higher on average.
Can I ask an AI lawyer for free?
Yes, you can ask a free chat about your contract or letter. We found no tests of how accurate free AI lawyers are, so a free answer needs the same check of citations on the portal.
Can AI replace an attorney?
Not in weighing evidence, says Svetlana Volodina, president of the Federal Chamber of Lawyers of Russia. She calls finding examples from case law a form of help, but in her words, weighing evidence must stay with a person. She said this at a conference on September 17, 2026, Advokatskaya Gazeta (the newspaper of Russia's Federal Chamber of Lawyers) reports.