The managing partner of a 14-person accounting firm sends a two-line email to the colleague who looks after IT: 'Can we train ChatGPT on our procedures manual and engagement letters? I want people to stop asking me where things are.'
It's a reasonable request, phrased the way nearly everyone phrases it. It also points toward the wrong technology. What the partner wants is an assistant that answers from the firm's own documents, shows where each answer came from and stays correct when the manual changes. That's a search problem with a writing step on top, and nothing needs to be trained.
Three things people mean by 'training it on our documents'
When someone asks to train an AI on company files, they usually mean one of three setups. They behave very differently once real people start asking questions.
| Approach | What actually happens | Good for | Weak at |
|---|---|---|---|
| Pasting or uploading into a chat | The text rides along with your question for that one conversation | A one-off question about one document | Many documents, repeat use, permissions |
| Retrieval (often called RAG) | Documents are indexed; each question triggers a search, and the best passages are handed to the model to answer from | Policies, manuals, FAQs and files that change | Questions your documents don't cover |
| Fine-tuning | The model's internal weights are adjusted using example questions and answers you supply | A consistent tone, format or narrow repeated task | Changing facts, citing sources, removing a document later |
RAG stands for retrieval-augmented generation. The term comes from a 2020 research paper that paired a language model with a separate, searchable index of documents (Wikipedia, in that case), so the knowledge lived outside the model and could be updated without retraining it. Business assistants are simpler than that research system, but the idea is the same: the model's general skill with language, plus a lookup into your material at the moment of the question.
If you've already uploaded files to a custom GPT or a ChatGPT project, you've been using this kind of setup, not training. OpenAI's documentation for ChatGPT Enterprise describes loading part of the text straight into the conversation and searching the rest through a private index when someone asks a question. Microsoft 365 Copilot works from the files and email you already have access to, and Gemini in Google Workspace can ground answers in Drive and Gmail.
In every case the model itself is unchanged. What differs is the plan you're on, because that decides whether your uploads can be used for training and how long they're kept.
How retrieval works, step by step
Say a staff accountant asks the firm's assistant: 'Do we charge extra if a client sends documents after March 15?' Here is what happens in the few seconds before an answer appears.
Flowchart
What happens when someone asks a retrieval-based assistant a question
- StartStaff accountant asks about the fee for documents received after March 15
- StepSystem checks which folders this person is allowed to open
- StepIt searches only those folders and pulls the five or six best-matching passages
- DecisionDo the passages actually answer the question?NoA personAssistant says it can't find this; the question goes to the partner, who adds the answer to the manualYes
- StepModel writes a short answer using only those passages
- DoneAnswer quotes section 6 of the current engagement letter, with a link to the file
Notice what isn't in that chain. Nothing about the engagement letter was baked into the model.
If the partner changes the late-documents fee tomorrow, the next answer reflects it as soon as the updated file is re-indexed. If a document is removed, it stops appearing. And because the answer quotes its source, anyone can check it in one click.
Retrieval doesn't make an assistant infallible. The search can pull the wrong passage, and the model can still misread a good one. That's why answers should quote their sources, and why we wrote a separate guide on reducing made-up answers.
Why fine-tuning is rarely the answer
Fine-tuning has real uses. If you want every draft letter in your firm's house style, or a model that fills in the same structured summary thousands of times, training it on good examples can help. As a way to store facts, it's a poor fit.
- It goes stale. A fine-tuned model reflects the examples it saw on the day it was trained. A fee change means another training run.
- It can't show its work. It doesn't point to the page an answer came from, so nobody can check it quickly.
- There's no simple undo. If a client file shouldn't have been included, you can't easily remove just that document.
- It ignores permissions. Whatever went into training can surface for anyone who uses the model.
Microsoft's documentation for fine-tuning in its Foundry platform says it 'doesn't replace retrieval for current information.' For a firm that wants answers from its manuals and files, retrieval is the default, and fine-tuning is an occasional extra for style.
Before and after
Training a model on the manual vs searching the manual at question time
Fine-tuned on the manual
- 01Manual turned into thousands of example questions and answers
- 02Model trained once; the manual is now blended into its weights
- 03Staff ask a question and get an answer with no source
- 04The late-documents fee changes in January
- 05Model keeps giving the old fee until someone retrains it
ResultFluent, unverifiable and out of date
Retrieval over the manual
- 01Manual indexed in short sections, each with a title and date
- 02Staff ask; the search finds the matching section
- 03Answer quotes that section and links to it
- 04The late-documents fee changes in January and the file is updated
- 05Next answer quotes the new fee
ResultAnswers anyone can check, that follow the document
Four checks before any client document goes in
For a law or accounting firm, the technical choice matters less than where the documents go and who can read them.
1. Which plan's data terms apply
Consumer and business AI plans are treated differently. As of November 2026, OpenAI says data from its business plans and API is excluded from training by default, while personal Free, Plus and Pro accounts have a training setting that's on unless the user turns it off. Anthropic's consumer terms leave the training decision to each Free, Pro and Max user and don't cover its commercial products.
Google says Workspace content isn't used to train its generative models without permission, and Microsoft says Copilot prompts and Microsoft 365 data aren't used to train its foundation models. These terms change often, so read the current version for the exact plan you'd use; staff on personal accounts are the usual gap.
2. Who can see what
A retrieval assistant should search only what the person asking is already allowed to open. Microsoft's Copilot documentation says it only surfaces data a user has at least view permission for. That's why a sharing cleanup comes first: an assistant will dutifully find the salary spreadsheet someone shared with the whole firm years ago.
3. Your professional duties
Tax preparers are limited by Section 7216 in how they use and disclose return information, and the FTC Safeguards Rule requires them to keep a written information security plan that covers their service providers. Lawyers have ABA Formal Opinion 512, which applies the duties of competence, confidentiality and supervision, among others, to generative AI. Neither rules out a well-built assistant, but both expect you to know where client data goes. That's general information, not legal advice, and the details vary by state.
4. Keeping sources current
Give every document an owner and a review date, and archive old versions instead of leaving them beside new ones. Two engagement letter templates in the same folder means two possible answers. Our guide to building the knowledge base behind an assistant goes deeper.
A one-week starting plan
- 01Monday: Collect 30 questions staff asked a partner or manager in the last month.
- 02Tuesday: Find the document that answers each one. Questions with no written answer are gaps to fill, not jobs for AI.
- 03Wednesday: Check sharing settings on those folders, and confirm which AI plan and data terms you'd be using.
- 04Thursday: Load only those documents into a retrieval tool, such as a project or custom GPT on a business ChatGPT plan, Copilot or Gemini pointed at one folder, or a purpose-built assistant, and ask all 30 questions.
- 05Friday: Mark each answer right and sourced, right but unsourced, wrong, or properly declined. Fix wrong answers in the document, not in the assistant's instructions.
If that week goes well, widen the document set slowly. If you'd rather have someone build and test it with you, see how we approach custom AI assistants; the first call is 30 minutes of practical answers with no hard sell.
Sources
- Lewis et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (NeurIPS 2020), arXiv
- Microsoft Learn: Fine-tuning in Microsoft Foundry
- Microsoft Learn: Retrieval-augmented generation (RAG) in Azure AI Search
- Microsoft Learn: Data, Privacy, and Security for Microsoft Copilot
- OpenAI Help Center: How your data is used to improve model performance
- OpenAI: Business data privacy, security, and compliance
- OpenAI Help Center: Optimizing file uploads in ChatGPT Enterprise
- OpenAI Help Center: Projects in ChatGPT
- Anthropic: Updates to Consumer Terms and Privacy Policy
- Google Workspace Help: Generative AI in Google Workspace Privacy Hub
- IRS: Section 7216 information center
- FTC: FTC Safeguards Rule, What Your Business Needs to Know
- American Bar Association: Formal Opinion 512, Generative Artificial Intelligence Tools (July 29, 2024)