searchi/docs/SPEECH-INTEGRATION.md

Local spoken passages

Google EmbeddingGemma 2 remains the semantic model for text, images, audio and video. An isolated local whisper.cpp worker adds recognized text to the same Google index. It does not replace raw audio evidence or introduce a hosted embedding backend.

The service covers an audio track with contiguous windows of at most thirty seconds, overlapping by two seconds. Live Silero detection resets per window; multilingual base recognition uses DTW token points. This differs from the historical whole-source diagnostic in SPEECH-DTW.md. Its 6/8 timing score cannot be carried over to this implementation.

Recognized passages retain source start/end points and exact recognized UTF-8. A point span can have zero duration. The UI starts playback at the estimated start and explains recognition uncertainty in the small match detail. A source point is not a verified complete utterance boundary.

Raw Google media and recognized speech have independent completion policies. Adding or retrying recognition reuses compatible raw vectors. Optional speech failures preserve raw search evidence and report partial coverage. Excerpts require current source bytes/revisions and the issued recognized passage; same-stat overwrites invalidate them. CLI, UI and MCP use this shared behavior.

Desktop setup downloads and verifies approximately 149 MB of base/VAD model data in addition to Google assets. Google query readiness stays independent. The native worker is shipped in an isolated runtime directory. Normal packages require both acceleration and CPU support; Windows VM execution is CPU only. Automatic retries use source revision and speech policy, preventing generation publication from creating an endless retry loop.

Each native call uses one admitted background CPU thread. Detailed indexing metrics separate admission, decoding, VAD, model loading, ASR, transport and fallback cost. A running native call cannot be preempted by a foreground query on a one-thread budget. Raw media decoding and ANN work are outside this inference budget.

Evidence in this source change

• Shared lifecycle, passage bounds, partial failures, provenance and policy tests passed. The complete workspace test command passed with genuine-model tests explicitly ignored pending their dedicated run.
• Two genuine CPU FFmpeg tests passed: a two-second audio origin, an internal timestamp gap, seeks and tails preserve the source timeline.
• Desktop Rust: 48 tests and strict Clippy passed. Frontend: 62 tests and formatting passed, including a stable partial-speech retry key.
• The actual CLI speech setup verified/reused the pinned base and Silero cache.
• Existing atomic staging tests passed; Windows package tests reject missing, changed, stale and internal CPU-only speech runtimes before replacing an existing archive.

Checks still required before delivery

• Current CPU native worker English/German context reuse and Unicode paths.
• Actual shared-service ASR plus Google lifecycle integration.
• Frozen independent four-file/ten-query and separate seam evaluation.
• Normal hardware-detecting runtime staging and Windows package startup.
• Publish/install the verified Windows update.
• Native playback and graphical interaction, actual GPU execution, and other OS runtime checks.

The independent fixture has no publisher phrase timestamps. It can measure retrieval and negative controls, but cannot establish precise word-level timing accuracy. Music relevance and general no-answer rejection remain separate open quality work; diagnostic integration alone does not raise the media rating.

Genuine CPU functional proofs

The internal v3 worker SHA is 299c1b784eac46b31d228453a399e1378f327fc9ed5e34ab950c804da33e433a; its inventory SHA is 4904995685a5e55e0a790d9cc409dc8458e39c30cda4097c5d70bf75cb5c976c. English/German tests relocated the worker and real models to paths containing München, 測試 and 🚀. Each repeated recognition returned stable bounded passages, and the second call reported zero additional model-load time. These checks selected Auto in the Rust client with SEARCHI_FORCE_CPU=1, so native execution was explicitly CPU. They do not validate dynamic acceleration loading.

The actual service lifecycle test passed in 113.50 seconds. Adding recognition, an injected optional failure on policy upgrade, and a genuine retry all retained byte-identical raw Google vectors with zero additional raw-audio encodes. Recognized results, exact verified excerpts, stable policy/source retry identity and the retained Google query reader were checked. This uses a query derived from recognition to test the contract; it is not an independent relevance score. Raw log .searchi/qa/service-speech-google-cpu-v3.log: 1292 bytes, SHA 170700adf9315ae15ae3ab5a7df6a77ab01dc5b776be179c861d2461cfc48c03.

The first normal auto compilation reached Vulkan setup but failed on Windows nested MSBuild path length. Its log is retained, SHA 59f1820d6ac6adac6f525d8084c442c1ed261753eb34fcce916dc157041f777c. An unchanged build in a shorter owned output directory is underway; no CPU-only distribution was substituted.