Skip to content

Before You Let an AI Scribe Into the Exam Room: 10 Questions to Ask

By USA AI Corp · · 5 min read

The short answer

Ask ten things before you sign: whether the vendor signs a BAA covering the scribe, how long it keeps audio and transcripts, whether it trains on them, how patient consent is captured under your state's recording law, how drafts reach your EHR, whether it handles your specialty, how it suggests codes, how accurate it is on your own visits, what sign-off looks like and whether it writes anything patients read.

The demo was flawless. The vendor's rep played both clinician and patient through a tidy, scripted visit for knee pain, and a well-organized note appeared ninety seconds later. Then you picture Thursday afternoon: a patient who speaks through an interpreter, a sixteen-year-old whose mother keeps answering for her, and a man who mentions chest tightness with his hand on the door handle.

An ambient AI scribe may cut after-hours charting for some clinicians, and the pilot below will show whether it does for yours. A demo only shows how a product handles the vendor's visit, not yours. These ten questions are meant to find the difference before you sign anything.

Privacy and data: questions 1 to 3

1. Will you sign a Business Associate Agreement that covers the scribe?

A scribe that records and transcribes visits creates and maintains protected health information for your practice, which makes the vendor a business associate under HIPAA. Get the BAA signed before any real visit is recorded, and confirm it covers the scribe product itself and binds the subcontractors that process audio, such as a speech-to-text or model provider.

2. What happens to the audio and transcript, and when?

Ask how long audio is kept after the note is drafted, whether transcripts are stored separately, and whether you can set or shorten those periods. Some practices want audio deleted as soon as the clinician signs; others want a short window to settle questions about what was said. Either way, the choice should be yours and written into the contract.

3. Do you use our recordings or notes to train or improve your models?

HIPAA allows a business associate to use patient information only as its contract permits or the law requires. If the vendor wants to use your visits to improve its product, even in de-identified form, that should be spelled out in the agreement and accepted by you, not added later in a terms-of-service update.

4. How does the patient consent, and where is that recorded?

Recording-consent laws vary by state, and several require everyone in a private conversation to agree. Washington's statute, for example, requires the consent of all persons engaged in a private conversation before it is recorded. Ask how the product prompts staff to ask, where the patient's answer is documented, and how quickly recording stops if the patient declines or a family member objects. This is general information, not legal advice; confirm your state's rules with counsel.

5. How do drafts get into our EHR?

Copying from a separate window is a different product from a draft that lands in the right section of the right encounter. Ask for a demonstration in your EHR, with your templates, and ask what happens when the integration fails mid-clinic.

6. Does it handle our specialty and our patients?

A scribe tuned on primary care may stumble over dermatology lesion descriptions or behavioral health nuance. Ask about specialty templates, visits in languages other than English, interpreters, several people talking at once, and sensitive topics a patient may not want in the note.

Flowchart

One visit with an ambient scribe, from consent to signature

  1. StartPatient checks in for a 2:40 PM visit
  2. StepMedical assistant explains the scribe in one sentence and asks permission
  3. DecisionDoes the patient agree to recording?
    No
    A personScribe stays off; the clinician documents the usual way, no questions asked
    Yes
  4. StepScribe records the visit and sends the audio to the vendor, under the BAA
  5. StepVendor's speech-to-text and AI models draft the note in the clinician's template
  6. A personClinician edits the draft, checks any suggested codes against the note and signs
  7. DoneAudio deleted on the schedule your contract sets
The patient decides whether the scribe listens, and the clinician decides what the note says.

Accuracy and coding: questions 7 to 9

7. How does it suggest billing codes?

If the product proposes visit levels or diagnosis codes, ask how. Each suggestion should point to the sentence in the note that justifies it, and the clinician should accept or reject it with one click. A tool that nudges visit levels upward without documentation to support them creates billing risk for your practice, not for the vendor. CMS's Evaluation and Management Services guide is a useful reference for what the documentation has to show.

8. How accurate is it on our visits, not the demo?

Ask for a pilot on your own patients and measure it yourself. The two-week scorecard below is a simple way to do it.

9. What do review and sign-off look like?

Every note is a draft until a clinician signs it. Check that AI-drafted text is easy to tell apart, that edits are quick and that nothing is filed to the chart without a clinician's action.

Before and after

Choosing a scribe from the demo vs from your own visits

Demo-day decision

  1. 01Vendor runs a scripted, single-problem visit
  2. 02Note looks clean in the vendor's template
  3. 03Retention questions answered on the call, not in writing
  4. 04Contract signed and rolled out to every clinician
  5. 05Problems discovered one real visit at a time

ResultA product chosen for a visit you'll never have

Two-week pilot

  1. 01BAA signed before any real visit is recorded
  2. 02Two or three clinicians use it on real visits, with consent
  3. 03Each draft scored for errors, omissions and edit time
  4. 04Hard visits (interpreter, several problems) included on purpose
  5. 05Rollout decided from the scorecard

ResultA decision based on your patients and your EHR

A short pilot on real visits surfaces problems a scripted demo never will.

Patient-facing text and the pilot: question 10

10. Does it write anything the patient will read?

Some scribes also draft after-visit summaries or portal replies. In California, AB 3030 says that when generative AI writes a patient message about clinical information, the message must carry an AI disclaimer and directions for reaching a human, unless a licensed or certified clinician has read and reviewed it first. Ask whether the product supports that disclaimer where needed, and keep clinician review as the default either way.

For the pilot, give two or three clinicians two weeks and score every AI-drafted note:

Pilot scorecard for each AI-drafted note
MeasureHow to record it
Added detailsCount of statements the patient or clinician never said: wrong medication, dose, side, history
OmissionsCount of clinically important points discussed but missing from the draft
Edit timeMinutes from opening the draft to signing it
Code suggestionsAccepted, changed or rejected, with a one-line reason
Hard visitsInterpreter, multi-problem and sensitive visits, scored separately
Clinician verdictWould you sign this with light edits? Yes or no

A scribe that saves time but adds a medication the patient never mentioned is not ready. Added details are the failure that matters most, so count them separately from omissions and review every one with the vendor.

Put four people on the evaluation: a clinician who will use it daily, your practice manager, whoever handles coding and billing, and whoever is responsible for privacy. Each will hear different things in the vendor's answers. Watch for these red flags:

  • The BAA is offered only after you sign the main contract.
  • Retention is described as 'industry standard' with no number of days.
  • Accuracy is a single figure from the vendor's own testing, with no description of the visits behind it.
  • Consent is left entirely to you, with no prompt or record in the product.
  • Code suggestions can't be traced to a sentence in the note.
  • There's no clear way to export or delete your data when you leave.
  • The contract is silent on who is responsible when a drafted error reaches a signed note, or lets the vendor change prices or auto-renew without notice.

If you'd like help running the pilot and setting a scribe up in your own templates, see how we work with medical practices. For patient-facing chat on your website, our guide to HIPAA and patient chatbots covers the rules from the other side.

Sources

Common questions

Quick answers.

Is an AI scribe HIPAA compliant?

No product is compliant on its own. Your use of it is compliant when the vendor signs a Business Associate Agreement covering the scribe, safeguards are in place, access is limited and retention follows your policies. Get the BAA before any real visit is recorded.

Do patients need to agree before an AI scribe records the visit?

Recording laws vary by state, and several require everyone in the conversation to consent. Asking every patient, documenting the answer and switching the scribe off when someone declines is the simplest safe default. Confirm your state's law with counsel.

How do you evaluate an ambient AI scribe?

Run a two-week pilot with two or three clinicians on real visits. Score every draft for added details, omissions, edit time and code suggestions, and include hard visits such as interpreted or multi-problem appointments. Decide from the scorecard, not the demo.

Can an AI scribe suggest billing codes?

Many can. Each suggestion should be traceable to specific documentation in the note, and the clinician should accept or reject it. A visit level the note doesn't support is a billing risk for the practice.

Put it to work

Custom AI chatbots & assistants

Custom AI chatbots and assistants grounded in your policies, documents and systems: website chat for customers and an internal assistant your team trusts.

See how it works

Most relevant for doctors.

Keep reading

Ready when the work is.

Show us what is slowing your business down.

Book a call

Not ready to talk?

Written brief

Skip the call.
Send a brief.

A few sentences is plenty. We read every brief and reply by email within one business day.

Try it free. If you love it, keep it. If it's not for you, we take it back, no questions asked.

We only use this to reply to you. Privacy policy