searchi/docs/SPEECH-INTEGRATION.md

Local spoken passages

Google EmbeddingGemma 2 remains the semantic model for text, images, audio and video. An isolated local whisper.cpp worker adds recognized text to the same Google index. It does not replace raw audio evidence or introduce a hosted embedding backend.

The service covers an audio track with contiguous windows of at most thirty seconds, overlapping by two seconds. Live Silero detection resets per window; multilingual base recognition uses DTW token points. This differs from the historical whole-source diagnostic in SPEECH-DTW.md. Its 6/8 timing score cannot be carried over to this implementation.

Recognized passages retain source start/end points and exact recognized UTF-8. A point span can have zero duration. The UI starts playback at the estimated start and explains recognition uncertainty in the small match detail. A source point is not a verified complete utterance boundary.

Raw Google media and recognized speech have independent completion policies. Adding or retrying recognition reuses compatible raw vectors. Optional speech failures preserve raw search evidence and report partial coverage. Excerpts require current source bytes/revisions and the issued recognized passage; same-stat overwrites invalidate them. CLI, UI and MCP use this shared behavior.

Desktop setup downloads and verifies approximately 149 MB of base/VAD model data in addition to Google assets. Google query readiness stays independent. The native worker is shipped in an isolated runtime directory. Normal packages require both acceleration and CPU support; Windows VM execution is CPU only. Automatic retries use source revision and speech policy, preventing generation publication from creating an endless retry loop.

Each native call uses one admitted background CPU thread. Detailed indexing metrics separate admission, decoding, VAD, model loading, ASR, transport and fallback cost. A running native call cannot be preempted by a foreground query on a one-thread budget. Raw media decoding and ANN work are outside this inference budget.

Evidence in this source change

• Shared lifecycle, passage bounds, partial failures, provenance and policy tests passed. The complete workspace test command passed with genuine-model tests explicitly ignored pending their dedicated run.
• Two genuine CPU FFmpeg tests passed: a two-second audio origin, an internal timestamp gap, seeks and tails preserve the source timeline.
• Desktop Rust: 48 tests and strict Clippy passed. Frontend: 62 tests and formatting passed, including a stable partial-speech retry key.
• The actual CLI speech setup verified/reused the pinned base and Silero cache.
• Existing atomic staging tests passed; Windows package tests reject missing, changed, stale and internal CPU-only speech runtimes before replacing an existing archive.

Checks still required before delivery

• Current CPU native worker English/German context reuse and Unicode paths.
• Actual shared-service ASR plus Google lifecycle integration.
• Frozen independent four-file/ten-query and separate seam evaluation.
• Normal hardware-detecting runtime staging and Windows package startup.
• Publish/install the verified Windows update.
• Native playback and graphical interaction, actual GPU execution, and other OS runtime checks.

The independent fixture has no publisher phrase timestamps. It can measure retrieval and negative controls, but cannot establish precise word-level timing accuracy. Music relevance and general no-answer rejection remain separate open quality work; diagnostic integration alone does not raise the media rating.

Genuine CPU functional proofs

The internal v3 worker SHA is 299c1b784eac46b31d228453a399e1378f327fc9ed5e34ab950c804da33e433a; its inventory SHA is 4904995685a5e55e0a790d9cc409dc8458e39c30cda4097c5d70bf75cb5c976c. English/German tests relocated the worker and real models to paths containing München, 測試 and 🚀. Each repeated recognition returned stable bounded passages, and the second call reported zero additional model-load time. These checks selected Auto in the Rust client with SEARCHI_FORCE_CPU=1, so native execution was explicitly CPU. They do not validate dynamic acceleration loading.

The actual service lifecycle test passed in 113.50 seconds. Adding recognition, an injected optional failure on policy upgrade, and a genuine retry all retained byte-identical raw Google vectors with zero additional raw-audio encodes. Recognized results, exact verified excerpts, stable policy/source retry identity and the retained Google query reader were checked. This uses a query derived from recognition to test the contract; it is not an independent relevance score. Raw log .searchi/qa/service-speech-google-cpu-v3.log: 1292 bytes, SHA 170700adf9315ae15ae3ab5a7df6a77ab01dc5b776be179c861d2461cfc48c03.

The first normal auto compilation reached Vulkan setup but failed on Windows nested MSBuild path length. Its log is retained, SHA 59f1820d6ac6adac6f525d8084c442c1ed261753eb34fcce916dc157041f777c. An unchanged build in a shorter owned output directory is underway; no CPU-only distribution was substituted.

The shorter-path normal auto build subsequently completed. Worker SHA 0b716fa17e8216ed46c7755d15acab20f658254b99780c7a68bab03d765c55bc, inventory SHA 0e116c604141af147bd91736ec0722a0660dcda8c9da008e494c464753f4e5a1. It contains private Vulkan/CPU backends, loader and matching Microsoft runtime files. The current normal artifact passed genuine repeated English recognition from Unicode paths with CPU forced. Fifty owned-process samples observed the CPU backend and no Vulkan modules; sampling alone does not prove absence at every instant. The pinned CPU-only loading branch complements that observation. See runtime evidence.

Frozen independent retrieval

The four-file fixture retained both real English/German recordings, instrumental music and silence, plus all ten frozen queries. Six positive queries found a relevant file first, with recall@5 also 6/6. Music and silence produced zero accepted recognized passages. All four absent queries still returned files. The two readings share translated subject matter; this is a small file-retrieval test, not broad media accuracy or verified passage/word timing.

Every expected window input and published citation passed source validation, including four windows across the two separate artificial seam sources. Transcription errors, partial boundary phrases and overlap duplicates remain in the evidence. Concept substring coverage is not a word-error score. Root independently recomputed the exact recorded scorer output.

The complete report is 174868 bytes, SHA 79ecb75d31c0193642c81862fd5476ad47adca518d4959e436bdd472cc281798. It uses the internal v3 CPU artifact and does not establish normal auto runtime quality parity or GPU performance. Details and the retained comparison are in the evaluator.