From da2a8e6cbbc7535e9c3201bf7fcf46fac126ca6e Mon Sep 17 00:00:00 2001 From: karl Date: Tue, 13 Oct 2026 07:31:26 +0200 Subject: [PATCH] Document local transcript alignment implementation and evidence gates --- docs/SPEECH-ALIGNMENT.md | 168 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 168 insertions(+) create mode 100644 docs/SPEECH-ALIGNMENT.md diff --git a/docs/SPEECH-ALIGNMENT.md b/docs/SPEECH-ALIGNMENT.md new file mode 100644 index 0000000..001b970 --- /dev/null +++ b/docs/SPEECH-ALIGNMENT.md @@ -0,0 +1,168 @@ +# Local spoken-passage alignment + +Status: researched implementation proposal, 2026-10-13. No transcription worker, +download integration, or ASR quality result is implemented by this document. +Google EmbeddingGemma 2 remains the production semantic model. + +## Why add a transcript lane + +[The frozen media evaluation](MEDIA-QUALITY.md) found the correct spoken-topic +file but achieved **0/8 precise timestamp hits** with production 30-second audio +windows; shorter windows achieved 1/8. A window's beginning is not the beginning +of the relevant utterance. Preserve raw Google audio/frame retrieval for sounds, +music and visual scenes; supplement it with locally recognized speech, timestamped +passages and real Google text vectors. Transcripts are fallible recognition output, +not verified quotations or confidence scores. + +## Runtime decision and first proof + +Use a **separate native whisper.cpp worker**, with a small bounded protocol and +Rust orchestration. Keep ASR outside the desktop process and outside the existing +Google worker: the two upstream ggml revisions must not share loaded libraries or +global symbols accidentally. Package ASR libraries in their own directory. The +first proof should compare multilingual **base** and **small** on the existing NASA +speech fixture, then held-out English/German recordings. Base is a download/cost +candidate, not an already accepted quality default; do not use English-only `.en` +weights for a bilingual product. + +Upstream whisper.cpp supports Windows/Linux/macOS, CPU and Metal/Vulkan backends; +its API accepts 16 kHz float PCM and returns segment times. Word/token alignment +and DTW are explicitly experimental. Start with sentence/segment timing, measure +the actual labelled point hits, and enable DTW only if a measured refinement is +needed. [Runtime documentation](https://github.com/ggml-org/whisper.cpp/blob/master/README.md), +[C API](https://github.com/ggml-org/whisper.cpp/blob/master/include/whisper.h). + +whisper-rs exposes the same timestamp controls, but embedding its native library +in the UI gives poorer crash isolation. It is a valid worker implementation option +after checking its bundled upstream revision and backend build flags; avoid mixing +it blindly with the pinned Google native stack. +[Rust API](https://docs.rs/whisper-rs/0.16.0/whisper_rs/struct.FullParams.html). + +No runtime build or genuine inference proof was run during this research. Obtain +the coordinator's CPU slot before building or transcribing in the shared VM. + +## Pinned assets and distribution + +Research resolved whisper.cpp v1.9.5 to commit +`d1be6fde11ac6e0407606b4e42fe72d34add8037` using the official GitHub tag API. +Model metadata was read from the Hugging Face blob API at repository revision +`5359861c739e955e79d9a303bcbc70fb988958b1`: + +| Candidate | Exact bytes | SHA-256 | +| --- | ---: | --- | +| `ggml-base.bin` | 147951465 | `60ed5bc3dd14eea856493d334349b405782ddcaf0028d4b5df4088345fba2efe` | +| `ggml-small.bin` | 487601967 | `1be3a9b2063867b937e64e2ec7483364a79917e157fa98c5d94b5c1fffea987b` | + +Use immutable download URLs of the form +`https://huggingface.co/ggerganov/whisper.cpp/resolve/5359861c739e955e79d9a303bcbc70fb988958b1/ggml-base.bin`. +These are **publisher metadata**, not hashes verified from downloaded bytes yet. +Commit a manifest only after a genuine download verifies exact size/SHA, model +loading and provenance. Never accept upstream `for-tests-*` empty models. +[Upstream model provenance](https://github.com/ggml-org/whisper.cpp/blob/master/models/README.md), +[blob metadata](https://huggingface.co/api/models/ggerganov/whisper.cpp?blobs=true). + +Ship the pinned worker in the normal package, automatically fetch verified weights +when speech indexing is enabled, and reuse the owned model cache. Downloads need +progress, cancellation, retries, partial-file recovery, quota checks, atomic rename +and offline status. No package-manager/compiler/Python requirement for users. +Whisper and whisper.cpp use MIT licenses; include both notices and converted-model +provenance in third-party attribution. Packaging review must inspect every bundled +dependency's license independently. +[Whisper license](https://github.com/openai/whisper/blob/main/LICENSE), +[whisper.cpp license](https://github.com/ggml-org/whisper.cpp/blob/master/LICENSE). + +Production worker builds include Metal on macOS or Vulkan on Windows/Linux and +CPU in the same package, with initialization failure retried on CPU. Backend DLLs +must load optionally, so an absent Vulkan loader cannot prevent CPU startup. +Record the backend actually used. All Windows VM and WSL proofs force CPU; physical +GPU speed/fallback and native macOS remain separate open gates. + +## Concrete integration contract + +1. Add `crates/searchi-speech` with `SpeechPolicy`, `SpeechModelManifest`, + `SpeechWorker`, `TranscriptSegment` and typed readiness/errors. Worker requests + carry bounded 16 kHz mono PCM, not arbitrary source paths. Reuse + `searchi-media::decode_audio`, which already produces that format and limits + each decode to 30 seconds. Keep any temporary files under the owned scratch + parent; never use a globally indexed temp folder. +2. Initial worker contract: one request at a time, 30-second maximum input, + finite PCM validation, bounded JSON output, explicit request id and worker/model + fingerprint. Return segment `start_ms`, `end_ms`, text and detected language; + reject out-of-range/nonmonotonic timestamps and oversized text. Parent timeout, + cancellation and owned process termination are mandatory. Do not log PCM/text. +3. `crates/searchi-service/src/media.rs` currently loops `audio_windows`, encodes + raw audio and inserts an `audio` row at `window.start_ms`. Add transcript work + beside that loop, retaining raw vectors. One ASR session is serialized; acquire + the shared background inference budget, use one CPU thread, and yield between + windows. ASR cannot preempt an already executing call. Decoder/extraction CPU + remains a separate explicitly measured budget. +4. Decode continuous windows with a bounded overlap. Convert local ASR times to + absolute media times once. Deduplicate only overlapping recognized segments + using time and normalized text; keep original text separately. Never concatenate + whole movies in RAM. Proposed limits (to validate): maximum 3-hour source, + 1800 windows, 10000 segments/file, 4096 UTF-8 bytes/segment and 16 MiB total + recognized text/file. Exceeding a limit reports partial/unready status rather + than claiming complete indexing. Empty/non-speech windows are valid results. +5. Add versioned transcript persistence in `searchi-service/src/store`: file id, + original file SHA-256, `file_change_revision`, stream index, absolute start/end, + recognized text, language, model/runtime/policy fingerprint and completion state. + Add a transcript FTS table with bounded candidates. Do not put unverified + transcript text into the existing original-document provenance fields. +6. Embed accepted timestamped passages with **Google text embeddings**, stored + as `transcript` chunks with recognized excerpt and timestamp source location. + Existing `SourceLocation { kind: "timestamp_ms", number: start_ms }` covers + opening/seek; an optional end bound can be added compatibly for previews. + Agent excerpts must distinguish recognized speech from source-document text. +7. Use a distinct completed-source key (`speech`, source SHA, ASR model SHA, + upstream runtime commit, stream selection, decoding/resampling version, + language/decoding/alignment/window/dedup policy, Google fingerprint). Same bytes + may reuse complete recognized segments/vector bytes after rename with current + citations. Changed policy cannot reuse stale results. Resume per-window staging + without falsely marking an incomplete source complete. +8. Revalidate SHA and captured change revision before publication. Native media + hints immediately invalidate old transcripts even with unchanged size/mtime; + deletion/missing sources suppress transcript FTS and semantic candidates before + limits. Guard writes with the captured revision, mirroring existing probe guards. + Prepare ANN before atomic generation activation; preserve the active generation + on ASR failure and keep other files' keyword/raw media search usable. + +## Retrieval and UI behavior + +Fuse bounded transcript FTS, transcript Google vectors, document/code vectors and +raw media candidates by ranked retrieval, preserving exact filename/metadata +priority and strict filters before candidate limits. Do not compare heterogeneous +raw scores as confidence. Do not route every query containing “speech” exclusively +to audio: code about speech detection and related documents remain candidates. +Avoid adding extra general-query embedding calls or repeated full eligibility +scans. Segment-level top hits should survive file deduplication with their matching +excerpt and precise timestamp. + +Show the recognized passage and timestamp in the detail preview, with a small +“Recognized speech” annotation; keep why-matched collapsed. Speech model setup and +indexing have honest progress/error states. Ordinary search stays local; optional +external Q&A requires the existing explicit endpoint/excerpt consent. Files, +transcripts, worker errors and inferred content never become executable instructions. + +## Acceptance sequence + +- [ ] Build isolated pinned CPU worker and verify downloaded base/small bytes. +- [ ] Decode the real NASA fixture, produce actual ASR JSON and manually inspect + recognized text against independent captions; record cost, peak private memory, + segment errors and timestamps. No invented transcript enters the proof. +- [ ] Compare the original eight frozen speech queries: correct file, ranking, + segment overlap, strict point hits and absolute-time conversion, separately from + overall media recall. Existing score stays 4/10 until a real rerun justifies change. +- [ ] Add held-out bilingual accents, long speech, leading silence, music-only, + non-speech, corrupt streams and absent topics; freeze labels before inference. + Hallucinated speech on silence/music is a measured failure, not silently accepted. +- [ ] Verify lifecycle: overwrite with same stats, rename, deletion, crash/resume, + changed model/policy, cancellation, stale publication and corrupted downloads. +- [ ] Measure foreground Google search during ASR background work, first input, + cold download/model load, warm p95, decoder/ASR/Google/storage stages separately. +- [ ] Verify UI, CLI/MCP recognized citations and filters through shared APIs. +- [ ] Validate normal package dependency closure and missing-driver CPU fallback + on each native OS; execute physical GPU tests outside this Windows VM. + +Next implementation slice: isolated worker + manifest verification + genuine NASA +proof. Keep the initial runtime experiment separate from schema/ranking changes +until it establishes that recognized segments actually fix the measured problem.