On March 1, 2023, a New York law firm filed a brief in a personal injury case against the airline Avianca. Among its citations were six court decisions, including one called Varghese v. China Southern Airlines, that did not exist. One of the lawyers had found them using ChatGPT.
The detail worth remembering comes from the judge's later opinion. The lawyer had asked ChatGPT directly, 'Is Varghese a real case,' and 'Are the other cases you provided fake.' ChatGPT replied that its authorities were real and could be found on Westlaw and LexisNexis.
In June 2023, Judge P. Kevin Castel of the Southern District of New York sanctioned the two lawyers and their firm, imposing a $5,000 penalty and requiring letters to each judge falsely named as the author of a fake opinion. He also wrote that there is 'nothing inherently improper about using a reliable artificial intelligence tool for assistance,' but that lawyers have a gatekeeping role to make sure their filings are accurate.
The lesson for any business isn't to avoid AI. It's that asking the model whether it's sure is not a check. The check has to come from outside the model: a source a person can open and read. (The case is told here as a cautionary example, not as legal advice, and professional rules on AI use differ from state to state.)
Why a business AI makes things up
A language model writes by predicting likely next words. Most of the time that produces accurate, fluent text. When the model lacks the facts, it still produces fluent text, and fluent text sounds sure of itself.
NIST, the federal standards agency, calls this 'confabulation' in its 2024 profile of generative AI risks. It notes that models can also produce made-up citations and reasoning that make a wrong answer more convincing. NIST's own example comes from health care: a confabulated summary of patient information could lead a doctor to the wrong diagnosis. In a law firm, the equivalent is a deadline or a citation.
The risk is highest where a wrong answer costs the most. That's why the fixes below are about the system around the model, not about hunting for a more truthful model.
Six controls that keep answers honest
1. Answer only from your sources
Connect the assistant to the documents that hold your answers (protocols, policies, templates and, where access rules allow, the matter or chart) and instruct it to answer only from what it retrieves. Answering only from your sources does the most. It reduces errors without eliminating them: in a preregistered Stanford evaluation, legal research tools from LexisNexis and Thomson Reuters built this way still hallucinated on between 17% and 33% of test queries, fewer than a general chatbot but far from zero. Patient charts belong only with a vendor under a Business Associate Agreement; our guide to using AI with client data covers those conditions.
2. Quote and link every answer
Each answer should show the passage it relied on, with a link to the document. A quote turns checking from a research task into a ten-second glance. An answer that can't point to a passage should be treated as unsupported.
3. Make 'I don't know' an approved answer
Write the refusal into the design. When the retrieved passages don't address the question, the assistant says so and offers a path to a person. Many made-up answers come from systems that were never given permission to say nothing.
4. Set a confidence threshold
Retrieval systems score how closely each passage matches the question. Below a threshold you choose, the assistant doesn't answer; it sends the question to staff along with the passages it found. Start strict and loosen only when testing shows it's safe.
Keep in mind what the score measures: how similar a passage is to the question, not whether it's true or current. An outdated policy can match perfectly, which is why the threshold works alongside quotes and review, not instead of them.
5. Test with real questions before launch
Use questions people actually asked, taken from your inbox, phone notes and staff, not questions written to make a demo shine. Include questions the assistant should decline.
6. Review misses every week
Log every question, answer, source and refusal. Once a week, someone who knows the subject reads the refusals, the low-confidence answers and a random sample of the rest, and fixes problems in the source document. Five patterns are worth looking for:
- Refusals on common questions: a sign a document is missing or poorly titled.
- Answers quoting an old version: archive the outdated file so search can't find it.
- Two sources that disagree: the owner decides which is right, and the other goes.
- Answers that drift beyond the quote: tighten the instruction to stay within the passage.
- Questions that should never be automated: add them to the always-route-to-a-person list.
Flowchart
How a grounded assistant decides whether to answer
- StartParalegal asks when the response to discovery in the Ortiz matter is due
- StepAssistant searches the Ortiz matter file and the firm's docketing notes
- DecisionIs the best match score above the threshold?NoA personIt shows what it found and sends the question to the docketing clerkYes
- DecisionDoes the passage state the deadline for this matter?NoA personIt says no docketed date was found and flags the responsible attorneyYes
- StepIt answers, quoting the docket entry with a link
- DoneParalegal checks the quote; the attorney still owns the deadline
Before and after
The same question, two kinds of assistant
General-purpose chatbot
- 01Asked when a discovery response is due
- 02Answers from general training, with no access to the file
- 03Gives a confident, generic deadline
- 04Offers no source, so the answer reads as authoritative
ResultPlausible, unverifiable, possibly wrong
Grounded assistant
- 01Searches the matter file and docket notes
- 02Finds the docket entry, or says it can't
- 03Quotes the entry and links to it
- 04Sends missing or low-confidence answers to a person
ResultRight with a source, or an honest 'not found'
A test plan you can run this week
Before an assistant answers anyone, run it against a fixed set of real questions and score the results. The same set becomes your regression test every time documents or settings change.
- 01Collect 50 real questions from the last two months: roughly 35 your documents should answer and 15 they shouldn't.
- 02For each, write the expected answer and its exact source, or 'should decline.'
- 03Run all 50 and score every answer with the table below.
- 04Fix each wrong answer at the source: a missing document, an outdated version or an unclear passage.
- 05Re-run the full set after every fix. Launch only when no high-stakes question gets a confident wrong answer.
| Result | What it means | What to do |
|---|---|---|
| Correct, with source | Matches the expected answer and quotes the right passage | Nothing |
| Correct, no source | Right answer with nothing to verify it against | Count it as a failure; fix retrieval or the document |
| Declined correctly | Your sources don't cover it, and it said so | Nothing, or write a document if the question is common |
| Declined wrongly | The answer was in your documents but it missed it | Improve the document's title, structure or wording |
| Wrong | A confident answer that doesn't match the source | Stop and trace it; this is the failure that matters |
A medical practice can run the same plan for staff-facing questions such as prior authorization rules, scheduling protocols and billing policies. Clinical content stays behind clinician review: a drafted visit summary or patient reply stays a draft until a clinician has signed off.
What stays with a person
None of these controls remove the need for judgment. Decide in writing which answers the assistant may give on its own and which it may only draft. In a law firm, deadlines, citations and anything resembling advice are drafts for a lawyer. In a medical practice, anything clinical goes to a clinician.
If your sources disagree with each other, fix that first; our guide to building a knowledge base for an assistant explains how.
Our post on keeping people in the loop covers where to draw the line, and our custom AI assistants are built around the six controls above.
Sources
- U.S. District Court, S.D.N.Y.: Mata v. Avianca, Inc., No. 22-cv-1461 (PKC), Opinion and Order on Sanctions (June 22, 2023)
- NIST AI 600-1: Artificial Intelligence Risk Management Framework, Generative Artificial Intelligence Profile (July 2024)
- Magesh et al., Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools, arXiv (published in the Journal of Empirical Legal Studies, 2025)