searchi/docs/SPEECH-ALIGNMENT.md

Local spoken-passage alignment

Status: researched implementation proposal, 2026-10-13. No transcription worker, download integration, or ASR quality result is implemented by this document. Google EmbeddingGemma 2 remains the production semantic model.

Why add a transcript lane

The frozen media evaluation found the correct spoken-topic file but achieved 0/8 precise timestamp hits with production 30-second audio windows; shorter windows achieved 1/8. A window's beginning is not the beginning of the relevant utterance. Preserve raw Google audio/frame retrieval for sounds, music and visual scenes; supplement it with locally recognized speech, timestamped passages and real Google text vectors. Transcripts are fallible recognition output, not verified quotations or confidence scores.

Runtime decision and first proof

Use a separate native whisper.cpp worker, with a small bounded protocol and Rust orchestration. Keep ASR outside the desktop process and outside the existing Google worker: the two upstream ggml revisions must not share loaded libraries or global symbols accidentally. Package ASR libraries in their own directory. The first proof should compare multilingual base and small on the existing NASA speech fixture, then held-out English/German recordings. Base is a download/cost candidate, not an already accepted quality default; do not use English-only .en weights for a bilingual product.

Upstream whisper.cpp supports Windows/Linux/macOS, CPU and Metal/Vulkan backends; its API accepts 16 kHz float PCM and returns segment times. Word/token alignment and DTW are explicitly experimental. Start with sentence/segment timing, measure the actual labelled point hits, and enable DTW only if a measured refinement is needed. Runtime documentation, C API.

whisper-rs exposes the same timestamp controls, but embedding its native library in the UI gives poorer crash isolation. It is a valid worker implementation option after checking its bundled upstream revision and backend build flags; avoid mixing it blindly with the pinned Google native stack. Rust API.

No runtime build or genuine inference proof was run during this research. Obtain the coordinator's CPU slot before building or transcribing in the shared VM.

Pinned assets and distribution

Research resolved whisper.cpp v1.9.5 to commit d1be6fde11ac6e0407606b4e42fe72d34add8037 using the official GitHub tag API. Model metadata was read from the Hugging Face blob API at repository revision 5359861c739e955e79d9a303bcbc70fb988958b1:

Candidate	Exact bytes	SHA-256
ggml-base.bin	147951465	60ed5bc3dd14eea856493d334349b405782ddcaf0028d4b5df4088345fba2efe
ggml-small.bin	487601967	1be3a9b2063867b937e64e2ec7483364a79917e157fa98c5d94b5c1fffea987b

Use immutable download URLs of the form https://huggingface.co/ggerganov/whisper.cpp/resolve/5359861c739e955e79d9a303bcbc70fb988958b1/ggml-base.bin. These are publisher metadata, not hashes verified from downloaded bytes yet. Commit a manifest only after a genuine download verifies exact size/SHA, model loading and provenance. Never accept upstream for-tests-* empty models. Upstream model provenance, blob metadata.

Ship the pinned worker in the normal package, automatically fetch verified weights when speech indexing is enabled, and reuse the owned model cache. Downloads need progress, cancellation, retries, partial-file recovery, quota checks, atomic rename and offline status. No package-manager/compiler/Python requirement for users. Whisper and whisper.cpp use MIT licenses; include both notices and converted-model provenance in third-party attribution. Packaging review must inspect every bundled dependency's license independently. Whisper license, whisper.cpp license.

Production worker builds include Metal on macOS or Vulkan on Windows/Linux and CPU in the same package, with initialization failure retried on CPU. Backend DLLs must load optionally, so an absent Vulkan loader cannot prevent CPU startup. Record the backend actually used. All Windows VM and WSL proofs force CPU; physical GPU speed/fallback and native macOS remain separate open gates.

Concrete integration contract

1. Add crates/searchi-speech with SpeechPolicy, SpeechModelManifest, SpeechWorker, TranscriptSegment and typed readiness/errors. Worker requests carry bounded 16 kHz mono PCM, not arbitrary source paths. Reuse searchi-media::decode_audio, which already produces that format and limits each decode to 30 seconds. Keep any temporary files under the owned scratch parent; never use a globally indexed temp folder.
2. Initial worker contract: one request at a time, 30-second maximum input, finite PCM validation, bounded JSON output, explicit request id and worker/model fingerprint. Return segment start_ms, end_ms, text and detected language; reject out-of-range/nonmonotonic timestamps and oversized text. Parent timeout, cancellation and owned process termination are mandatory. Do not log PCM/text.
3. crates/searchi-service/src/media.rs currently loops audio_windows, encodes raw audio and inserts an audio row at window.start_ms. Add transcript work beside that loop, retaining raw vectors. One ASR session is serialized; acquire the shared background inference budget, use one CPU thread, and yield between windows. ASR cannot preempt an already executing call. Decoder/extraction CPU remains a separate explicitly measured budget.
4. Decode continuous windows with a bounded overlap. Convert local ASR times to absolute media times once. Deduplicate only overlapping recognized segments using time and normalized text; keep original text separately. Never concatenate whole movies in RAM. Proposed limits (to validate): maximum 3-hour source, 1800 windows, 10000 segments/file, 4096 UTF-8 bytes/segment and 16 MiB total recognized text/file. Exceeding a limit reports partial/unready status rather than claiming complete indexing. Empty/non-speech windows are valid results.
5. Add versioned transcript persistence in searchi-service/src/store: file id, original file SHA-256, file_change_revision, stream index, absolute start/end, recognized text, language, model/runtime/policy fingerprint and completion state. Add a transcript FTS table with bounded candidates. Do not put unverified transcript text into the existing original-document provenance fields.
6. Embed accepted timestamped passages with Google text embeddings, stored as transcript chunks with recognized excerpt and timestamp source location. Existing SourceLocation { kind: "timestamp_ms", number: start_ms } covers opening/seek; an optional end bound can be added compatibly for previews. Agent excerpts must distinguish recognized speech from source-document text.
7. Use a distinct completed-source key (speech, source SHA, ASR model SHA, upstream runtime commit, stream selection, decoding/resampling version, language/decoding/alignment/window/dedup policy, Google fingerprint). Same bytes may reuse complete recognized segments/vector bytes after rename with current citations. Changed policy cannot reuse stale results. Resume per-window staging without falsely marking an incomplete source complete.
8. Revalidate SHA and captured change revision before publication. Native media hints immediately invalidate old transcripts even with unchanged size/mtime; deletion/missing sources suppress transcript FTS and semantic candidates before limits. Guard writes with the captured revision, mirroring existing probe guards. Prepare ANN before atomic generation activation; preserve the active generation on ASR failure and keep other files' keyword/raw media search usable.

Retrieval and UI behavior

Fuse bounded transcript FTS, transcript Google vectors, document/code vectors and raw media candidates by ranked retrieval, preserving exact filename/metadata priority and strict filters before candidate limits. Do not compare heterogeneous raw scores as confidence. Do not route every query containing “speech” exclusively to audio: code about speech detection and related documents remain candidates. Avoid adding extra general-query embedding calls or repeated full eligibility scans. Segment-level top hits should survive file deduplication with their matching excerpt and precise timestamp.

Show the recognized passage and timestamp in the detail preview, with a small “Recognized speech” annotation; keep why-matched collapsed. Speech model setup and indexing have honest progress/error states. Ordinary search stays local; optional external Q&A requires the existing explicit endpoint/excerpt consent. Files, transcripts, worker errors and inferred content never become executable instructions.

Acceptance sequence

• Build isolated pinned CPU worker and verify downloaded base/small bytes.
• Decode the real NASA fixture, produce actual ASR JSON and manually inspect recognized text against independent captions; record cost, peak private memory, segment errors and timestamps. No invented transcript enters the proof.
• Compare the original eight frozen speech queries: correct file, ranking, segment overlap, strict point hits and absolute-time conversion, separately from overall media recall. Existing score stays 4/10 until a real rerun justifies change.
• Add held-out bilingual accents, long speech, leading silence, music-only, non-speech, corrupt streams and absent topics; freeze labels before inference. Hallucinated speech on silence/music is a measured failure, not silently accepted.
• Verify lifecycle: overwrite with same stats, rename, deletion, crash/resume, changed model/policy, cancellation, stale publication and corrupted downloads.
• Measure foreground Google search during ASR background work, first input, cold download/model load, warm p95, decoder/ASR/Google/storage stages separately.
• Verify UI, CLI/MCP recognized citations and filters through shared APIs.
• Validate normal package dependency closure and missing-driver CPU fallback on each native OS; execute physical GPU tests outside this Windows VM.

Next implementation slice: isolated worker + manifest verification + genuine NASA proof. Keep the initial runtime experiment separate from schema/ranking changes until it establishes that recognized segments actually fix the measured problem.