Engineering

Private by design, and it learns your voice

July 22, 2026

The recording of a dental visit is the most sensitive artifact CareNotes ever handles, and the one it holds for the shortest time. This is the architecture that keeps the audio on the iPhone, runs speech recognition and speaker separation there, sends only text onward, and still gets better at recognizing one particular clinician's voice every time it is used.

There is a straightforward way to build an AI scribe: stream the microphone to a server, transcribe it there, run a language model, return a note. It is the easiest to ship and the hardest to defend. A server-side recording is protected health information at rest, under the HIPAA Security Rule's technical safeguards [1], with a retention policy, an access log, and a breach surface. It is also, for a practice in Auckland or Frankfurt, a cross-border transfer of a patient's voice.

CareNotes made the other choice early, and the work between April and July this year finished it. Speech-to-text runs on the phone. Speaker separation runs on the phone. The audio buffer is released once those two steps are done. What leaves the device is a text transcript, over TLS, to a model endpoint pinned to the practice's region. Along the way, the phone keeps a small statistical sketch of the clinician's voice, updates it after each recording, and since July syncs that sketch, never audio, through the account so that signing in on a new phone, iPhone or Android, picks up where the old one left off.

Why on-device speech recognition became practical

Five years ago this architecture would have meant shipping a weaker recognizer. The gap closed from two directions. OpenAI's Whisper showed that an encoder-decoder transformer trained on 680,000 hours of weakly supervised audio generalizes across accents, noise, and domains without fine-tuning [2], and its open weights made it portable. Then the mobile runtimes caught up: open-source projects such as WhisperKit compile those models to Core ML and run them on the Apple Neural Engine with latency and energy figures that make a full-visit transcription on an iPhone routine [3]. Apple's own SpeechAnalyzer, introduced in iOS 26, added a first-party offline engine with on-device language assets [4].

CareNotes uses more than one on-device engine, chosen per job by device capability and language. A Whisper-family model is the workhorse for recordings made on the phone. Apple's SpeechAnalyzer handles audio that arrives from the Apple Watch on iOS 26 and newer, and the app pre-warms it when a recording language is chosen so the first words are not lost to model loading. NVIDIA's Parakeet, a FastConformer model with a token-and-duration transducer decoder [5], covers the languages still marked Beta on recent hardware. All of them produce text on the device. None of them opens a socket.

The scheduler around them is strict about memory. Heavier engines run one job at a time; lighter ones may run a few in parallel on devices that can afford it. A recording in progress always preempts a transcription or diarization job, and the job is re-queued rather than dropped, because the one thing the app must never do is miss words while it is busy with a previous visit.

Speaker separation without a server

A transcript of a dental visit is only half-useful if it does not say who is speaking. The dentist's description of a lesion and the patient's description of their pain look the same on paper. CareNotes separates speakers in two layers, both on the phone.

Layer one: neural diarization

After a recording stops, the 16 kHz mono buffer the recognizer already holds is handed, without touching disk, to a diarization pipeline built on pyannote.audio [6]. Pyannote's approach is end-to-end neural segmentation over short windows, producing per-frame speaker activity, followed by speaker embedding and clustering to stitch local labels into global ones; the 2023 pipeline revision reformulated the segmentation objective as a powerset multi-class problem, which made overlapping speech, common when a patient answers mid-sentence, tractable [7]. The models are open-source, downloaded once on first use, and run through Core ML. There is no license server call and no per-device key.

Diarization is file-level post-processing, not streaming. We tried streaming and turned it off. Streaming labels have to commit early and cannot revise; file-level diarization sees the whole visit and can decide that the voice at minute two and the voice at minute nine are the same person. The cost is a short wait after the recording, which the app hides behind the note-generation step.

Layer two: which speaker is the clinician?

Neural diarization answers "how many speakers, and when does each speak." It does not know which one is the dentist. That is the job of the clinician voice profile, and it is the part that learns.

The profile is deliberately small. It is not an embedding and not a recording; it is a handful of summary statistics over acoustic features that are cheap to compute on every segment: fundamental frequency (pitch), RMS energy, and the first two formants F1 and F2, which track jaw opening and tongue position and are among the more speaker-specific properties of vowels.

For each of those features the profile keeps a mean and a standard deviation, plus a sample count and a confidence that grows with the number of visits it has learned from. That is the whole profile: a few dozen floating-point numbers.

Matching a segment to the profile is a weighted z-score distance. Each feature's deviation from the profile mean is divided by the profile's standard deviation for that feature, the groups are combined with fixed weights (pitch carries the most, formants the least), and a segment is accepted as the clinician if the combined distance falls under a threshold of roughly two weighted standard deviations. Standard deviations are floored so that a very consistent enrollment cannot produce a profile that rejects the same person on a day they have a cold.

z(feature) = |segment.feature − profile.mean| / profile.stddev
distance   = w_pitch · z(pitch) + w_energy · z(energy) + w_formant · mean(z(F1), z(F2))
clinician  = distance < threshold

A reader who knows the speaker-recognition literature will notice this is a long way from a modern x-vector system [8]. That is intentional. The profile has one job: among the two or three speakers the neural diarizer already found, pick the one that is the enrolled clinician. A low-dimensional, interpretable model is enough for that, it costs nothing at runtime, and it can be inspected when it is wrong. When it is wrong, the per-feature z-scores say why.

Enrollment, and learning from every visit

Enrollment takes about thirty seconds. The clinician reads four sentences aloud while the app highlights each word as it is recognized, so they can see the recognizer keeping up:

CareNotes mission is very simple. When doctors get their time back, they show up better for their patients. They show up better for their families, and themselves. That ripple effect is how we can transform healthcare from within.

From those segments the app computes the initial means and deviations. The enrollment audio is processed in memory and is not kept.

Then the profile keeps learning. After every recording is finalized, the segments the profile already accepts as the clinician are used to build a session profile, and the stored profile moves toward it with an exponential moving average:

clinician_segments = [s for s in recording if profile.matches(s)]
session_profile    = summarize(clinician_segments)
profile            = (1 − α) · profile + α · session_profile

An update is only committed when enough clinician segments were found to make the session statistics meaningful, which is when the app shows "Voice profile updated." This is the same idea as the MAP adaptation in classical GMM speaker verification [9], reduced to its moments: trust the prior, move a fixed fraction toward the evidence, and never let one unusual session dominate. With a small α a single session's influence decays geometrically over the following recordings, so a day of recording in a noisy room with a different microphone washes out on its own. Self-training has a known failure mode, drift toward whichever speaker is matched most often, and the acceptance threshold on the existing profile before an update is the guard against it: a patient's segments have to already look like the clinician to be used for learning.

One profile, every phone

Until July the profile lived in local storage on one device. A clinician who upgraded their iPhone started over, and one who used both an iPhone and an Android phone had two profiles that disagreed. Since July the profile is written to a small per-user record in the practice's regional database, scoped to the signed-in account.

The record holds the statistics described above as plain numbers, nothing else. There is no audio and no embedding to reverse. The phone keeps a cached copy for offline use and listens for changes, so a profile updated after a recording on one device is on the other within seconds of it coming online. A reset is modelled as deletion of the record, which every client treats as "no profile, re-enroll." The reader rejects malformed or out-of-range values rather than silently accepting a corrupted profile, because a bad profile fails quietly by labelling the patient as the clinician.

What leaves the phone, and where it goes

Being precise about this matters more than being reassuring. After transcription and diarization, the audio buffer is released and any temporary file is deleted. What remains is text: a transcript with clinician and patient speaker labels, and the clinician's own facts and preferences for note style.

That text is sent to a large language model to draft the note. The primary model is a Gemini model served through Google's Vertex AI, and the endpoint is chosen by the practice's region, so a New Zealand practice's transcripts are processed in Australia, a European practice's in the EU, and a US practice's in the United States, with Canadian, Brazilian, and Middle East stacks alongside. Google's data-governance terms for Vertex AI generative models state that prompts and responses are not used to train foundation models [10]; the model processing runs under an executed BAA with zero data retention, and CareNotes signs its own Business Associate Agreement with every practice before PHI is processed. Notes are generated in English and then translated on the same path if the clinician's note language is different; periodontal visits use a purpose-built generator rather than the general template.

Two things we do not claim. First, the browser path is different: the Chrome extension and Visits capture audio on a computer, not an iPhone, and process it under the terms in the privacy policy rather than the on-device pipeline described here. Second, "HIPAA compliant" describes how we operate, not a certificate anyone issues; the controls, subprocessors, retention periods, and the downloadable BAA are published in the Trust Center so a practice's compliance officer can check rather than take our word.

What the architecture buys

The honest summary is that keeping audio on the device is slower to build and makes the product better in ways that do not show up in a feature list. The recognizer runs during the visit with no connectivity requirement, which matters in basement operatories. There is no recording to subpoena, leak, or retain. The voice profile learns from every visit without any voice data ever reaching us, which is the only way we were comfortable building a feature that learns at all. And the same on-device engines are what make Carey's voice possible in the other direction.

The notes themselves, once drafted and signed, go on to the chart; that pipeline is in Tap once, it's in the chart. The diarizer's ability to tell the hygienist from the patient is also what keeps a patient's "three?" out of a perio chart.