VoiceDebrief iOS / articles / three-layers

Three layers: code, the model on the device, and agents

Kind: a position. From: the owner, 15 November 2026: "three levels of intelligence, or of execution ... what we run at each layer is the superpower." Written up by the Sherpa session. Every example links to what is built, which is the only page that says what exists; most of it is in test and has not been run on a device.

Three bands. Code on the device: records, keeps, decides states, counts. Apple's model on the device: transcribes, debriefs, labels, cleans, picks one action; free, private, small. Agents in a loop: the conductor drains lanes, files notes in vaults, replies, writes the app itself; capable, slow, costs per run. A voice note goes down through them: code records and keeps it, the local model transcribes and labels it, the conductor acts on it, and its reply comes back into the app.

The idea in one paragraph

Three things in this project can do work. The code in the app is exact, instant and free, and can only do what somebody wrote. Apple's models on the device (speech recognition and the Foundation Models framework) understand language, cost nothing per use and never send anything away, but are small. Agents (Claude sessions running in a loop, with the conductor's tools, lanes and vaults) can read, write, plan and build, but take minutes, cost money per run and see only what is sent to them. None is best. The skill is choosing, for each step of a job, the lowest layer that can do it well, and handing the result up only when it needs more.

Layer one: the code

What it is: the Swift in the app, and the scripts beside it. It holds the states, keeps the data and decides everything that has a rule.

What it does today:

Why here: it must be exact, it must work offline and instantly, and it must be the same every time. Bad at: anything that needs understanding words.

Layer two: Apple's model on the device

What it is: SpeechAnalyzer for the transcript and the Foundation Models framework for everything that reads or writes language, both running on the iPhone, iPad or Mac.

What it does today:

Why here: it is free per use, so it can run all the time ("I don't feel like I'm using minutes"); nothing leaves the device; it answers in about a second. Bad at: long contexts (4,096 tokens on an iPhone, 8,192 on an iPad), rambling notes (it once gave a task to "[Claude]" because the note addressed Claude by name), and anything that needs the world outside the device. So it is asked one small question at a time, with the code deciding what happens with the answer (the position of two decision engines).

Layer three: agents in a loop

What it is: Claude sessions, and the conductor that receives what the app sends, working with lanes, vaults and the repository.

What it does today:

Why here: it can read everything sent to it, plan, write code, call any tool and keep going. Bad at: speed (minutes, not seconds), cost (every run is paid for), privacy (it sees what it is sent, so what goes up is chosen and sealed), and presence (it is not on the device when the network is not).

How a task moves between the layers

A voice note for someone, end to end:

  1. Code. Tap to record; the note exists on disk at once; the level bars move; the end phrase is matched.
  2. Local model. The transcript is made on the device; labels (who, topic, urgency) can be filled in while speaking; a debrief is written.
  3. Code. The state moves to "waiting to send"; the note is sealed to the recipient's key; the send is retried until it goes, and the row says sent, with the time.
  4. Agents. The conductor drains the lane, files the note in a vault, acts on it (an issue, a reply, a change to the app) and writes the answer back.
  5. Code. The app pulls the vault and shows the reply under the note; the briefing counts it.

Each step is done at the lowest layer that can do it, and each handover is a file, a sealed message or a commit that can be seen. Nothing is left in a model's memory.

Where each thing belongs

CodeLocal modelAgents
Cost per usenonenonepaid per run
Privacyon the deviceon the devicesees what is sent
Speedinstantabout a secondminutes
Works offlineyesyesno
Good atrules, states, keeping, cryptoshort language tasks, one question at a timereading, planning, building, acting in the world
Bad atunderstanding wordslong context, rambling, the outside worldspeed, cost, being on the device
In this projectNoteStore, Briefing, VaultBook, VoiceCommandSpeechFileTranscriber, OnDeviceModel, Lab L02 and L04tools/conductor, the agents vault, the sessions

What follows from it