The demo was flawless. The vendor's rep played both clinician and patient through a tidy, scripted visit for knee pain, and a well-organized note appeared ninety seconds later. Then you picture Thursday afternoon: a patient who speaks through an interpreter, a sixteen-year-old whose mother keeps answering for her, and a man who mentions chest tightness with his hand on the door handle.
An ambient AI scribe may cut after-hours charting for some clinicians, and the pilot below will show whether it does for yours. A demo only shows how a product handles the vendor's visit, not yours. These ten questions are meant to find the difference before you sign anything.
Privacy and data: questions 1 to 3
1. Will you sign a Business Associate Agreement that covers the scribe?
A scribe that records and transcribes visits creates and maintains protected health information for your practice, which makes the vendor a business associate under HIPAA. Get the BAA signed before any real visit is recorded, and confirm it covers the scribe product itself and binds the subcontractors that process audio, such as a speech-to-text or model provider.
2. What happens to the audio and transcript, and when?
Ask how long audio is kept after the note is drafted, whether transcripts are stored separately, and whether you can set or shorten those periods. Some practices want audio deleted as soon as the clinician signs; others want a short window to settle questions about what was said. Either way, the choice should be yours and written into the contract.
3. Do you use our recordings or notes to train or improve your models?
HIPAA allows a business associate to use patient information only as its contract permits or the law requires. If the vendor wants to use your visits to improve its product, even in de-identified form, that should be spelled out in the agreement and accepted by you, not added later in a terms-of-service update.
Consent and the visit itself: questions 4 to 6
4. How does the patient consent, and where is that recorded?
Recording-consent laws vary by state, and several require everyone in a private conversation to agree. Washington's statute, for example, requires the consent of all persons engaged in a private conversation before it is recorded. Ask how the product prompts staff to ask, where the patient's answer is documented, and how quickly recording stops if the patient declines or a family member objects. This is general information, not legal advice; confirm your state's rules with counsel.
5. How do drafts get into our EHR?
Copying from a separate window is a different product from a draft that lands in the right section of the right encounter. Ask for a demonstration in your EHR, with your templates, and ask what happens when the integration fails mid-clinic.
6. Does it handle our specialty and our patients?
A scribe tuned on primary care may stumble over dermatology lesion descriptions or behavioral health nuance. Ask about specialty templates, visits in languages other than English, interpreters, several people talking at once, and sensitive topics a patient may not want in the note.
Flowchart
One visit with an ambient scribe, from consent to signature
- StartPatient checks in for a 2:40 PM visit
- StepMedical assistant explains the scribe in one sentence and asks permission
- DecisionDoes the patient agree to recording?NoA personScribe stays off; the clinician documents the usual way, no questions askedYes
- StepScribe records the visit and sends the audio to the vendor, under the BAA
- StepVendor's speech-to-text and AI models draft the note in the clinician's template
- A personClinician edits the draft, checks any suggested codes against the note and signs
- DoneAudio deleted on the schedule your contract sets
Accuracy and coding: questions 7 to 9
7. How does it suggest billing codes?
If the product proposes visit levels or diagnosis codes, ask how. Each suggestion should point to the sentence in the note that justifies it, and the clinician should accept or reject it with one click. A tool that nudges visit levels upward without documentation to support them creates billing risk for your practice, not for the vendor. CMS's Evaluation and Management Services guide is a useful reference for what the documentation has to show.
8. How accurate is it on our visits, not the demo?
Ask for a pilot on your own patients and measure it yourself. The two-week scorecard below is a simple way to do it.
9. What do review and sign-off look like?
Every note is a draft until a clinician signs it. Check that AI-drafted text is easy to tell apart, that edits are quick and that nothing is filed to the chart without a clinician's action.
Before and after
Choosing a scribe from the demo vs from your own visits
Demo-day decision
- 01Vendor runs a scripted, single-problem visit
- 02Note looks clean in the vendor's template
- 03Retention questions answered on the call, not in writing
- 04Contract signed and rolled out to every clinician
- 05Problems discovered one real visit at a time
ResultA product chosen for a visit you'll never have
Two-week pilot
- 01BAA signed before any real visit is recorded
- 02Two or three clinicians use it on real visits, with consent
- 03Each draft scored for errors, omissions and edit time
- 04Hard visits (interpreter, several problems) included on purpose
- 05Rollout decided from the scorecard
ResultA decision based on your patients and your EHR
Patient-facing text and the pilot: question 10
10. Does it write anything the patient will read?
Some scribes also draft after-visit summaries or portal replies. In California, AB 3030 says that when generative AI writes a patient message about clinical information, the message must carry an AI disclaimer and directions for reaching a human, unless a licensed or certified clinician has read and reviewed it first. Ask whether the product supports that disclaimer where needed, and keep clinician review as the default either way.
For the pilot, give two or three clinicians two weeks and score every AI-drafted note:
| Measure | How to record it |
|---|---|
| Added details | Count of statements the patient or clinician never said: wrong medication, dose, side, history |
| Omissions | Count of clinically important points discussed but missing from the draft |
| Edit time | Minutes from opening the draft to signing it |
| Code suggestions | Accepted, changed or rejected, with a one-line reason |
| Hard visits | Interpreter, multi-problem and sensitive visits, scored separately |
| Clinician verdict | Would you sign this with light edits? Yes or no |
A scribe that saves time but adds a medication the patient never mentioned is not ready. Added details are the failure that matters most, so count them separately from omissions and review every one with the vendor.
Put four people on the evaluation: a clinician who will use it daily, your practice manager, whoever handles coding and billing, and whoever is responsible for privacy. Each will hear different things in the vendor's answers. Watch for these red flags:
- The BAA is offered only after you sign the main contract.
- Retention is described as 'industry standard' with no number of days.
- Accuracy is a single figure from the vendor's own testing, with no description of the visits behind it.
- Consent is left entirely to you, with no prompt or record in the product.
- Code suggestions can't be traced to a sentence in the note.
- There's no clear way to export or delete your data when you leave.
- The contract is silent on who is responsible when a drafted error reaches a signed note, or lets the vendor change prices or auto-renew without notice.
If you'd like help running the pilot and setting a scribe up in your own templates, see how we work with medical practices. For patient-facing chat on your website, our guide to HIPAA and patient chatbots covers the rules from the other side.
Sources
- HHS: Business Associate Contracts (sample provisions)
- eCFR: 45 CFR 164.502, uses and disclosures of protected health information
- Washington State Legislature: RCW 9.73.030, intercepting, recording or divulging private communication
- California Department of Public Health: AFL 25-07, Assembly Bill 3030, Health Care Services: Artificial Intelligence
- California Legislative Information: AB 3030 (2024), Health care services: artificial intelligence
- CMS Medicare Learning Network: Evaluation and Management Services (MLN006764)