searchi/docs/SPEECH-ALIGNMENT.md

# Local spoken-passage alignment

Status: implementation proposal with an internal CPU diagnostic, 2026-10-13.
Production transcription, download integration and retrieval acceptance are open.
Google EmbeddingGemma 2 remains the production semantic model.

A bounded [CPU diagnostic harness](../tools/speech-eval/README.md) is now prepared.
Seven structural tests passed on Windows, including actual owned subprocess
failure, timeout, output overflow and retained diagnostics. Both candidate models
have now been downloaded and their exact bytes/SHA verified. Both loaded in the
internal experiment, whose truncated audio invalidated quality observations.
Corrected evaluation remains pending; structural tests establish no speech quality.

## Why add a transcript lane

[The frozen media evaluation](MEDIA-QUALITY.md) found the correct spoken-topic
file but achieved **0/8 precise timestamp hits** with production 30-second audio
windows; shorter windows achieved 1/8. A window's beginning is not the beginning
of the relevant utterance. Preserve raw Google audio/frame retrieval for sounds,
music and visual scenes; supplement it with locally recognized speech, timestamped
passages and real Google text vectors. Transcripts are fallible recognition output,
not verified quotations or confidence scores.

## Runtime decision and first proof

Use a **separate native whisper.cpp worker**, with a small bounded protocol and
Rust orchestration. Keep ASR outside the desktop process and outside the existing
Google worker: the two upstream ggml revisions must not share loaded libraries or
global symbols accidentally. Package ASR libraries in their own directory. The
first proof should compare multilingual **base** and **small** on the existing NASA
speech fixture, then held-out English/German recordings. Base is a download/cost
candidate, not an already accepted quality default; do not use English-only `.en`
weights for a bilingual product.

Upstream whisper.cpp supports Windows/Linux/macOS, CPU and Metal/Vulkan backends;
its API accepts 16 kHz float PCM and returns segment times. Word/token alignment
and DTW are explicitly experimental. Start with sentence/segment timing, measure
the actual labelled point hits, and enable DTW only if a measured refinement is
needed. [Runtime documentation](https://github.com/ggml-org/whisper.cpp/blob/master/README.md),
[C API](https://github.com/ggml-org/whisper.cpp/blob/master/include/whisper.h).

whisper-rs exposes the same timestamp controls, but embedding its native library
in the UI gives poorer crash isolation. It is a valid worker implementation option
after checking its bundled upstream revision and backend build flags; avoid mixing
it blindly with the pinned Google native stack.
[Rust API](https://docs.rs/whisper-rs/0.16.0/whisper_rs/struct.FullParams.html).

The initial research performed no inference. Subsequent coordinator-approved CPU
diagnostics are recorded below; production integration and acceptance remain open.

## Pinned assets and distribution

Research resolved whisper.cpp v1.9.5 to commit
`d1be6fde11ac6e0407606b4e42fe72d34add8037` using the official GitHub tag API.
Model metadata was read from the Hugging Face blob API at repository revision
`5359861c739e955e79d9a303bcbc70fb988958b1`:

| Candidate | Exact bytes | SHA-256 |
| --- | ---: | --- |
| `ggml-base.bin` | 147951465 | `60ed5bc3dd14eea856493d334349b405782ddcaf0028d4b5df4088345fba2efe` |
| `ggml-small.bin` | 487601967 | `1be3a9b2063867b937e64e2ec7483364a79917e157fa98c5d94b5c1fffea987b` |

Use immutable download URLs of the form
`https://huggingface.co/ggerganov/whisper.cpp/resolve/5359861c739e955e79d9a303bcbc70fb988958b1/ggml-base.bin`.
Both downloads were independently checked against these exact bytes/SHA values
on 2026-10-13. The clean runtime checkout resolves to the pinned commit; model
URLs, verified hashes and MIT notices are retained under ignored
`.searchi/speech/provisioning.json`. These hashes establish provisioning;
the subsequent internal runtime experiment does not establish release acceptance. Commit an
accepted distribution manifest only after model loading and runtime provenance
checks. Never accept upstream `for-tests-*` empty models.
[Upstream model provenance](https://github.com/ggml-org/whisper.cpp/blob/master/models/README.md),
[blob metadata](https://huggingface.co/api/models/ggerganov/whisper.cpp?blobs=true).

Ship the pinned worker in the normal package, automatically fetch verified weights
when speech indexing is enabled, and reuse the owned model cache. Downloads need
progress, cancellation, retries, partial-file recovery, quota checks, atomic rename
and offline status. No package-manager/compiler/Python requirement for users.
Whisper and whisper.cpp use MIT licenses; include both notices and converted-model
provenance in third-party attribution. Packaging review must inspect every bundled
dependency's license independently.
[Whisper license](https://github.com/openai/whisper/blob/main/LICENSE),
[whisper.cpp license](https://github.com/ggml-org/whisper.cpp/blob/master/LICENSE).

Production worker builds include Metal on macOS or Vulkan on Windows/Linux and
CPU in the same package, with initialization failure retried on CPU. Backend DLLs
must load optionally, so an absent Vulkan loader cannot prevent CPU startup.
Record the backend actually used. All Windows VM and WSL proofs force CPU; physical
GPU speed/fallback and native macOS remain separate open gates.

## Concrete integration contract

1. Add `crates/searchi-speech` with `SpeechPolicy`, `SpeechModelManifest`,
   `SpeechWorker`, `TranscriptSegment` and typed readiness/errors. Worker requests
   carry bounded 16 kHz mono PCM, not arbitrary source paths. Reuse
   `searchi-media::decode_audio`, which already produces that format and limits
   each decode to 30 seconds. Keep any temporary files under the owned scratch
   parent; never use a globally indexed temp folder.
2. Initial worker contract: one request at a time, 30-second maximum input,
   finite PCM validation, bounded JSON output, explicit request id and worker/model
   fingerprint. Return segment `start_ms`, `end_ms`, text and detected language;
   reject out-of-range/nonmonotonic timestamps and oversized text. Parent timeout,
   cancellation and owned process termination are mandatory. Do not log PCM/text.
3. `crates/searchi-service/src/media.rs` currently loops `audio_windows`, encodes
   raw audio and inserts an `audio` row at `window.start_ms`. Add transcript work
   beside that loop, retaining raw vectors. One ASR session is serialized; acquire
   the shared background inference budget, use one CPU thread, and yield between
   windows. ASR cannot preempt an already executing call. Decoder/extraction CPU
   remains a separate explicitly measured budget.
4. Decode continuous windows with a bounded overlap. Convert local ASR times to
   absolute media times once. Deduplicate only overlapping recognized segments
   using time and normalized text; keep original text separately. Never concatenate
   whole movies in RAM. Proposed limits (to validate): maximum 3-hour source,
   1800 windows, 10000 segments/file, 4096 UTF-8 bytes/segment and 16 MiB total
   recognized text/file. Exceeding a limit reports partial/unready status rather
   than claiming complete indexing. Empty/non-speech windows are valid results.
5. Add versioned transcript persistence in `searchi-service/src/store`: file id,
   original file SHA-256, `file_change_revision`, stream index, absolute start/end,
   recognized text, language, model/runtime/policy fingerprint and completion state.
   Add a transcript FTS table with bounded candidates. Do not put unverified
   transcript text into the existing original-document provenance fields.
6. Embed accepted timestamped passages with **Google text embeddings**, stored
   as `transcript` chunks with recognized excerpt and timestamp source location.
   Existing `SourceLocation { kind: "timestamp_ms", number: start_ms }` covers
   opening/seek; an optional end bound can be added compatibly for previews.
   Agent excerpts must distinguish recognized speech from source-document text.
7. Use a distinct completed-source key (`speech`, source SHA, ASR model SHA,
   upstream runtime commit, stream selection, decoding/resampling version,
   language/decoding/alignment/window/dedup policy, Google fingerprint). Same bytes
   may reuse complete recognized segments/vector bytes after rename with current
   citations. Changed policy cannot reuse stale results. Resume per-window staging
   without falsely marking an incomplete source complete.
8. Revalidate SHA and captured change revision before publication. Native media
   hints immediately invalidate old transcripts even with unchanged size/mtime;
   deletion/missing sources suppress transcript FTS and semantic candidates before
   limits. Guard writes with the captured revision, mirroring existing probe guards.
   Prepare ANN before atomic generation activation; preserve the active generation
   on ASR failure and keep other files' keyword/raw media search usable.

## Retrieval and UI behavior

Fuse bounded transcript FTS, transcript Google vectors, document/code vectors and
raw media candidates by ranked retrieval, preserving exact filename/metadata
priority and strict filters before candidate limits. Do not compare heterogeneous
raw scores as confidence. Do not route every query containing “speech” exclusively
to audio: code about speech detection and related documents remain candidates.
Avoid adding extra general-query embedding calls or repeated full eligibility
scans. Segment-level top hits should survive file deduplication with their matching
excerpt and precise timestamp.

Show the recognized passage and timestamp in the detail preview, with a small
“Recognized speech” annotation; keep why-matched collapsed. Speech model setup and
indexing have honest progress/error states. Ordinary search stays local; optional
external Q&A requires the existing explicit endpoint/excerpt consent. Files,
transcripts, worker errors and inferred content never become executable instructions.

## Acceptance sequence

- [ ] Build isolated pinned CPU worker and verify downloaded base/small bytes.
- [ ] Decode the real NASA fixture, produce actual ASR JSON and manually inspect
  recognized text against independent captions; record cost, peak private memory,
  segment errors and timestamps. No invented transcript enters the proof.
- [ ] Compare the original eight frozen speech queries: correct file, ranking,
  segment overlap, strict point hits and absolute-time conversion, separately from
  overall media recall. Existing score stays 4/10 until a real rerun justifies change.
- [ ] Add held-out bilingual accents, long speech, leading silence, music-only,
  non-speech, corrupt streams and absent topics; freeze labels before inference.
  Hallucinated speech on silence/music is a measured failure, not silently accepted.
- [ ] Verify lifecycle: overwrite with same stats, rename, deletion, crash/resume,
  changed model/policy, cancellation, stale publication and corrupted downloads.
- [ ] Measure foreground Google search during ASR background work, first input,
  cold download/model load, warm p95, decoder/ASR/Google/storage stages separately.
- [ ] Verify UI, CLI/MCP recognized citations and filters through shared APIs.
- [ ] Validate normal package dependency closure and missing-driver CPU fallback
  on each native OS; execute physical GPU tests outside this Windows VM.

The runtime and genuine retrieval diagnostics below inform future work; they do
not add a production speech feature. Further ASR/VAD research is deferred while
the usable Windows desktop takes priority. Revisit gating/alignment before schema
or ranking changes that would expose unreliable recognized speech to users.

## Actual internal CPU diagnostic, 2026-10-13

The clean pinned v1.9.5 source compiled with MSVC in **54.655 seconds**, jobs one,
static whisper/ggml, `GGML_NATIVE=OFF`; CUDA, Vulkan, Metal, BLAS and OpenMP were
explicitly disabled in the recorded CMake cache. This is an **internal CPU test
executable**, not a normal distribution or portability proof. The compiled CPU
variant uses AVX2; an older-CPU release has not been validated. The executable SHA
is `12c681b4bcd9c7679b647b9a041e121a2f9e72cb2a9c6f189f529ec8b79c36f8`.
Help flags were checked and actual runs pass `--no-gpu --threads 1 --processors 1`.
Dumpbin found Windows system/UCRT and MSVC runtime imports, with no external
ggml/whisper/GPU imports. Existing VM runtime availability is not a clean-machine
or older-Windows dependency-closure guarantee.

The initial bounded base experiment completed all five source files; the small
experiment was stopped by terminating only its owned process tree after a material
harness bug was identified. Both are classified **INVALID_TRUNCATED_AUDIO**.
`atrim=end_sample` was applied before explicit 16 kHz resampling: a planned
30-second window supplied approximately 10 seconds from 48 kHz sources or 10.884
seconds from 44.1 kHz sources. Recognition omissions and process costs from those
runs cannot establish complete-window model quality or throughput. Earlier draft
observations about base/small recognition are withdrawn; no ASR model default or
improved search score follows from these invalid experiments.

Actual downloaded NASA publisher captions are independently retained: 2204 bytes,
SHA `80d82def36547d95c512034deb9df9d3d0160579a43c8177e6ca988d3b02b04b`.
Their utterance intervals agree with the frozen v1 labels. Comparing recognized
text against these captions must be repeated after corrected decoding.
[NASA publisher captions](https://images-assets.nasa.gov/video/Artemis%20Origins/Artemis%20Origins.srt).
No independent human audio listening has been performed.

The decoder now explicitly resamples to 16000 Hz before trimming and resets sample
timestamps. Every decoded WAV records its frame count and rejects coverage outside
expected duration minus at most one millisecond. Actual CPU FFmpeg regressions
verified both 44.1 kHz and 48 kHz sources: a 30-second window has exactly 480000
samples and a one-second tail exactly 16000. Generated silent WAVs exercise
resampling mechanics only; they are not ASR quality fixtures. Ten tests passed in
1.406 seconds, including rejection of threefold truncation, bounded owned-child
errors/timeouts/output, retained diagnostics and continuation after invalid ASR
timestamp output. They do not establish speech recognition quality.

Historical raw/checkpoint hashes and invalid classification remain under ignored
`.searchi/speech/invalid-truncated-audio.json`; model bytes and actual raw outputs
are excluded from Git. The owned sequential session 20331 is terminal with exit 1
following intentional invalid-experiment termination. A subsequent corrected run
also detected a source-timeline issue: NASA's container ends at 85.333333 seconds,
but its primary audio ends at 85.262 seconds. Session 6691 failed closed on the
72 ms tail discrepancy, preserving its completed checkpoint. The harness now
uses the actual primary audio stream's duration/start/index, absolute seeking
with `-seek_timestamp 1`, and explicit stream mapping. Missing stream duration
is an actionable error; container duration is never silently substituted.

Twelve tests passed in 1.937 seconds after that correction, including actual
FFmpeg mux/decode with a longer video, audio offset 0.125 seconds and container
offset five seconds. The audio timeline was 5.125–36.125 seconds; its one-second
tail decoded 15997 samples, within the unchanged one-millisecond rounding bound.
The zero-start 44.1/48 kHz tests still decoded exactly 480000/16000 samples.

## Corrected base candidate evidence

The corrected base evaluation completed all five assets with zero timestamp or
sample-coverage validation failures. Report SHA:
`9e33984afac9480452f506070dee0156fcc8d7f0d6882c8f5feeeaf5cae57de2`.
Executed frozen harness SHA:
`16e29bb70d3e1eeaad1f5b8ec223bff1a6f359215bcf8dabe52512a451855346`.

| Source | Windows | Recognized segments | ASR process seconds | Decode seconds |
| --- | ---: | ---: | ---: | ---: |
| NASA | 3 | 19 | 23.322 | 0.542 |
| Piano | 9 | 9 | 210.720 | 2.167 |
| Violin | 1 | 1 | 4.471 | 0.212 |
| Bark | 1 | 0 | 4.373 | 0.113 |
| Roaring effect | 1 | 1 | 4.732 | 0.111 |

These are observed serial CPU process times including model startup, not warmed
throughput or a controlled hardware comparison. NASA's decoded windows contained
480000, 480000 and 404192 samples, covering the actual primary audio range.
Manual comparison with publisher captions finds the Artemis funding and Mars
statements recognized, but the initial Moon-choice statement is merged into a
segment starting at zero and the orange-soil exchange is omitted. Recognition
does not prove Google ranking or a precise timestamp success rate.

The instrumental piano emitted an unrelated water phrase and repetitive numbers
or punctuation in several windows. Music descriptions also appear. These must
remain competing retrieved candidates in the next diagnostic; filtering them
by knowledge of labels would hide failure. The source's published instrumental
description supports a hallucination concern, but independent audio listening
remains open. The current report field `quality_failed=false` means only that
structural timestamp/sample validation passed; **it does not certify speech or
search quality**. No model default is accepted.

## Corrected small candidate evidence

Owned sequential session 89200 finished with exit zero after evaluating all five
assets, base then small. Both used the same frozen corrected harness and verified
model bytes. The complete small report SHA is
`1660c95c71c9b8b1cb3b77f8f1ec422fb0954f2aacd2c16ea773901c0cfbe099`.

| Source | Windows | Valid recognized segments | ASR process seconds | Decode seconds |
| --- | ---: | ---: | ---: | ---: |
| NASA | 3 | 19 | 86.250 | 0.601 |
| Piano | 9 | 8 | 1176.426 | 2.399 |
| Violin | 1 | 1 | 16.045 | 0.162 |
| Bark | 1 | 0 | 16.701 | 0.112 |
| Roaring effect | 1 | 1 | 25.082 | 0.113 |

Small emitted repetitive inferred words in non-English languages on the
instrumental piano, with substantial CPU cost. Two actual timestamp outputs
failed validation: the 14.454-second piano tail returned a 0–30-second segment,
and the 2.039-second bark window returned a 0–2.5-second segment. Decoded sample
counts were correct (231254 and 32618 respectively); these are ASR timestamp
failures, not the historical resampling bug. Raw JSON is retained; no timestamp
was clamped. The report's `complete=true` establishes complete evaluation,
while `quality_failed=true` records failed structural validation.

NASA's small recognition retained funding and Mars passages but misrecognized
the orange-soil words and still produced coarse segment boundaries. No numeric
timestamp retrieval improvement follows from caption inspection. The small
ranker may retain diagnostic results, but its reduced valid-window pool must be
`quality_score_eligible=false`; the scorer refuses an accepted numeric quality
score for incomplete or reduced competitor pools. Thirteen harness regressions,
including real FFmpeg coverage/timeline checks and this scoring gate, passed in
the coordinator's verified-FFmpeg run (2.100 seconds).

Final process inspection found no owned ASR/runner processes; the CPU slot was
released for genuine Google transcript ranking. The current run removed its
temporary decoded WAV/JSON. An older scratch directory from intentionally stopped
invalid session 20331 remains ignored: automatic approval review rejected native
PowerShell recursive cleanup with only “blocked by policy” as its reason. No
alternate deletion method was attempted.

Neither ASR model is an accepted product default. Production Google transcript integration,
speech activity gating, tighter alignment, hallucination control, German speech,
resource scheduling and native distribution still require evidence. Publish raw
Google media search before completing derived transcription so slow ASR cannot
block initial semantic readiness. Transcript publication needs its own bounded
completed-source work and atomic eligibility update. The production media rating
remains 4/10.

## Actual Google transcript retrieval diagnostic

The coordinator's owned session 2984 completed base and small ranking with exit
zero. The shared real Google adapter ran on CPU, one thread: document encoding
used the normal indexing role (2048-token context), then the document process was
dropped before opening the query role (512 tokens). Both reported
`google/embeddinggemma-2`, 768 dimensions and `b11461-no-logits-v1`. The diagnostic
runner was a Debug build with SHA
`be096c6dfe158b991520c060ca83478bdd0cd5c34d64fab4a2413d8b01de8f15`;
the actual native Google worker SHA was
`c01cb3ec13cfca722ee768e8350e1399a8b9a4d14a24a35adc3473a5cfc82a73`.

The base experiment ranked all **30 valid recognized segments from all five
sources**, preserving music/sound descriptions and hallucinated text as
competitors. Actual Google tokenizer windows preserved original UTF-8 source
ranges. Segment ranking used exact cosine and maximum over its token windows;
the frozen labels did not select or rewrite candidates. The eight unchanged
English/German NASA queries produced:

| Metric | Result |
| --- | ---: |
| Correct file at rank one | 8/8 |
| Strict timestamp start hit at rank one | 4/8 |
| Segment overlap at rank one | 6/8 |

Both Moon-choice queries selected the merged segment starting at zero, before
the labelled 1.551–3.603-second utterance. Both orange-soil queries selected an
unrelated passage starting at 24.320 seconds because the recognized exchange
was missing. Funding queries selected 43.920 seconds and Mars queries selected
60 seconds, within the independently frozen intervals. Correct-file success
does not establish precise spoken-passage localization.

The complete base Google report SHA is
`2fbc860f992c3ec561f54137e30744b00265455792e360010ab1fc78ab25173a`;
the accepted diagnostic scoring artifact SHA is
`8f92c8eeed41db7e78be0e188500e441d53b4fa49fa8c0353a77df74643ce895`.
Raw artifacts remain ignored under `.searchi/speech`. This is **direct exact-cosine
retrieval of recognized text**, not production ANN/hybrid retrieval of raw audio.
The historical production 0/8 audio-window result is useful motivation but is
not an equivalent head-to-head experiment; no production improvement or score
change is claimed. Eight correlated queries about four topics in one English
video do not validate German speech, held-out relevance or abstention.

The small diagnostic also completed, with report SHA
`aca99d36b3bdf013b1a1074c33f7232de1c1b79af62d00236d53267ce510c9c1`.
Its two invalid ASR windows (piano tail and bark) were explicitly excluded:
`complete=true`, `quality_score_eligible=false`, `excluded_window_count=2`.
Diagnostic ranks are retained, but **no accepted numeric quality score** is
published for that reduced competitor pool. Neither model is adopted. Next steps
are speech gating/alignment proof, representative negative controls, actual
production integration and an equivalent shared-pipeline rerun; the media rating
remains 4/10.