VoiceDebrief iOS / sgit-app / analysis

What is in a file: an analysis layer for vaults

Proposed, 15 November 2026, at the owner's request. None of this is built. What is built is the reality file. This page maps how the sgit app could add facts, transcripts and descriptions to each file in a vault, using Apple's models on the device only.

The rule that makes it survive moves: everything hangs off the content hash

The app already keeps two things beside every photo and video it uploads, filed by the file's content hash, which depends only on its bytes:

Analysis follows the same rule, one file per analyser:

.sgit/analysis/<hash>/transcript.json     what is said, with times
.sgit/analysis/<hash>/transcript.vtt      the same as captions a player can show
.sgit/analysis/<hash>/vision.json         labels, text on screen, faces counted, codes read
.sgit/analysis/<hash>/summary.json        a title, a few sentences, tags

What follows from that:

Each record says what made it, so that a better model later can redo it without guessing:

{
  "format": "sgit.ai/file-analysis/v1",
  "contentHash": "…",
  "analyser": "apple.speech.transcript",
  "analyserVersion": "iOS 26.1",
  "locale": "en-GB",
  "madeAt": "2026-11-15T20:00:00Z",
  "result": { … }
}

What Apple's models can do with a video, on the device

QuestionApple frameworkWhat comes outNotes
What is said?Speech: SpeechAnalyzer with SpeechTranscriber (the VoiceDebrief app uses it today, in SpeechFileTranscriber.swift)A transcript with a time for each phrase; captionsThe video's sound is taken out first (AVAssetExportSession to .m4a) and handed to the same code as a voice note. Runs on the device, offline once the language model is downloaded
What is in it?Vision: image classification, on a frame every few seconds and at each cutLabels with confidence ("crowd", "stage", "car", "food") and when they appearCheap: frames are decoded at low resolution
What text is on screen?Vision: text recognitionSlides, badges, signs ("KNOW THE REACH", "ATTENDEE Nikos")Useful for search; the same works on photos
Who is in it?Vision: face detectionHow many faces, and whenCounting only. Recognising who a person is would be a separate, opt-in decision for the owner
Any codes?Vision: barcode detectionQR codes and barcodes read in the frameThe attendee badge in the vault has one
Which frame is best?Vision: aesthetics scores, saliencyA better thumbnail than the first frameAlso improves the gallery
What is it about?Foundation Models (the on-device language model)A short title, two or three sentences, tagsIt takes text, not pictures, in the version the app builds against: it works from the transcript, labels and on-screen text above. One question at a time, through a function call, as the project's standing position asks

On the server side: no. Apple's Private Cloud Compute is not something an app can call for its own work, and this project has no backend of its own by design. Everything above runs on the iPhone, iPad or Mac that has the file.

Where and when it runs

  1. At upload, while the plain copy is still there. This is how thumbnails are made today. It costs no download. Transcribing a long video takes minutes and battery, so it would be queued and run while the device is charging.
  2. Afterwards, on request, for files that came in another way (the web Vault, the sgit command, an older build). "Analyse this folder" fetches each file once, analyses it, keeps the results and lets the copy go. The screen says how much it will fetch first ("5 videos, 2.1 GB").
  3. Never twice. Before analysing, the app looks for .sgit/analysis/<hash>/. If another device or a teammate already did it, the work is done.

The same job fixes something visible today: videos added without a stored thumbnail show a plain film icon, because making one means fetching the whole file. A first version of "Analyse this folder" that only makes the missing thumbnails and facts would fix the owner's movies folder.

What a person sees

Build order (proposed)

  1. Missing thumbnails and facts, made on request for files that have none. Small, and fixes the film icons.
  2. Transcripts of videos, reusing the VoiceDebrief transcriber; captions in the viewer; the transcript in the (i) page.
  3. Vision labels and on-screen text for photos and frames of videos.
  4. A summary and tags from the on-device language model, working from 2 and 3.
  5. Search across the vault, on the device.

Questions for the owner