146 lines
8.1 KiB
Markdown
146 lines
8.1 KiB
Markdown
# Local spoken passages
|
|
|
|
Google EmbeddingGemma 2 remains the semantic model for text, images, audio and
|
|
video. An isolated local whisper.cpp worker adds recognized text to the same
|
|
Google index. It does not replace raw audio evidence or introduce a hosted
|
|
embedding backend.
|
|
|
|
The service covers an audio track with contiguous windows of at most thirty
|
|
seconds, overlapping by two seconds. Live Silero detection resets per window;
|
|
multilingual base recognition uses DTW token points. This differs from the
|
|
historical whole-source diagnostic in [SPEECH-DTW.md](SPEECH-DTW.md). Its 6/8
|
|
timing score cannot be carried over to this implementation.
|
|
|
|
Recognized passages retain source start/end points and exact recognized UTF-8.
|
|
A point span can have zero duration. The UI starts playback at the estimated
|
|
start and explains recognition uncertainty in the small match detail. A source
|
|
point is not a verified complete utterance boundary.
|
|
|
|
Raw Google media and recognized speech have independent completion policies.
|
|
Adding or retrying recognition reuses compatible raw vectors. Optional speech
|
|
failures preserve raw search evidence and report partial coverage. Excerpts
|
|
require current source bytes/revisions and the issued recognized passage;
|
|
same-stat overwrites invalidate them. CLI, UI and MCP use this shared behavior.
|
|
|
|
Desktop setup downloads and verifies approximately 149 MB of base/VAD model
|
|
data in addition to Google assets. Google query readiness stays independent.
|
|
The native worker is shipped in an isolated runtime directory. Normal packages
|
|
require both acceleration and CPU support; Windows VM execution is CPU only.
|
|
Automatic retries use source revision and speech policy, preventing generation
|
|
publication from creating an endless retry loop.
|
|
|
|
Each native call uses one admitted background CPU thread. Detailed indexing
|
|
metrics separate admission, decoding, VAD, model loading, ASR, transport and
|
|
fallback cost. A running native call cannot be preempted by a foreground query
|
|
on a one-thread budget. Raw media decoding and ANN work are outside this
|
|
inference budget.
|
|
|
|
## Evidence in this source change
|
|
|
|
- Shared lifecycle, passage bounds, partial failures, provenance and policy
|
|
tests passed. The complete workspace test command passed with genuine-model
|
|
tests explicitly ignored pending their dedicated run.
|
|
- Two genuine CPU FFmpeg tests passed: a two-second audio origin, an internal
|
|
timestamp gap, seeks and tails preserve the source timeline.
|
|
- Desktop Rust: 48 tests and strict Clippy passed. Frontend: 62 tests and
|
|
formatting passed, including a stable partial-speech retry key.
|
|
- The actual CLI speech setup verified/reused the pinned base and Silero cache.
|
|
- Existing atomic staging tests passed; Windows package tests reject missing,
|
|
changed, stale and internal CPU-only speech runtimes before replacing an
|
|
existing archive.
|
|
|
|
## Checks still required before delivery
|
|
|
|
- [x] Current CPU native worker English/German context reuse and Unicode paths.
|
|
- [x] Actual shared-service ASR plus Google lifecycle integration.
|
|
- [x] Frozen independent four-file/ten-query and separate seam evaluation.
|
|
- [ ] Normal hardware-detecting runtime staging and Windows package startup.
|
|
- [ ] Publish/install the verified Windows update.
|
|
- [ ] Native playback and graphical interaction, actual GPU execution, and
|
|
other OS runtime checks.
|
|
|
|
The independent fixture has no publisher phrase timestamps. It can measure
|
|
retrieval and negative controls, but cannot establish precise word-level timing
|
|
accuracy. Music relevance and general no-answer rejection remain separate open
|
|
quality work; diagnostic integration alone does not raise the media rating.
|
|
|
|
## Genuine CPU functional proofs
|
|
|
|
The internal v3 worker SHA is
|
|
`299c1b784eac46b31d228453a399e1378f327fc9ed5e34ab950c804da33e433a`;
|
|
its inventory SHA is
|
|
`4904995685a5e55e0a790d9cc409dc8458e39c30cda4097c5d70bf75cb5c976c`.
|
|
English/German tests relocated the worker and real models to paths containing
|
|
`München`, `測試` and `🚀`. Each repeated recognition returned stable bounded
|
|
passages, and the second call reported zero additional model-load time. These
|
|
checks selected `Auto` in the Rust client with `SEARCHI_FORCE_CPU=1`, so native
|
|
execution was explicitly CPU. They do not validate dynamic acceleration loading.
|
|
|
|
The actual service lifecycle test passed in 113.50 seconds. Adding recognition,
|
|
an injected optional failure on policy upgrade, and a genuine retry all retained
|
|
byte-identical raw Google vectors with zero additional raw-audio encodes.
|
|
Recognized results, exact verified excerpts, stable policy/source retry identity
|
|
and the retained Google query reader were checked. This uses a query derived
|
|
from recognition to test the contract; it is not an independent relevance score.
|
|
Raw log `.searchi/qa/service-speech-google-cpu-v3.log`: 1292 bytes, SHA
|
|
`170700adf9315ae15ae3ab5a7df6a77ab01dc5b776be179c861d2461cfc48c03`.
|
|
|
|
The first normal auto compilation reached Vulkan setup but failed on Windows
|
|
nested MSBuild path length. Its log is retained, SHA
|
|
`59f1820d6ac6adac6f525d8084c442c1ed261753eb34fcce916dc157041f777c`.
|
|
An unchanged build in a shorter owned output directory was started; no CPU-only
|
|
distribution was substituted.
|
|
|
|
The shorter-path normal auto build subsequently completed. Worker SHA
|
|
`0b716fa17e8216ed46c7755d15acab20f658254b99780c7a68bab03d765c55bc`,
|
|
inventory SHA
|
|
`0e116c604141af147bd91736ec0722a0660dcda8c9da008e494c464753f4e5a1`.
|
|
It contains private Vulkan/CPU backends, loader and matching Microsoft runtime
|
|
files. The current normal artifact passed genuine repeated English recognition
|
|
from Unicode paths with CPU forced. Fifty owned-process samples observed the
|
|
CPU backend and no Vulkan modules; sampling alone does not prove absence at
|
|
every instant. The pinned CPU-only loading branch complements that observation.
|
|
See [runtime evidence](../tools/speech-runtime/README.md).
|
|
|
|
## Frozen independent retrieval
|
|
|
|
The four-file fixture retained both real English/German recordings, instrumental
|
|
music and silence, plus all ten frozen queries. Six positive queries found a
|
|
relevant file first, with recall@5 also 6/6. Music and silence produced zero
|
|
accepted recognized passages. **All four absent queries still returned files.**
|
|
The two readings share translated subject matter; this is a small file-retrieval
|
|
test, not broad media accuracy or verified passage/word timing.
|
|
|
|
Every expected window input and published citation passed source validation,
|
|
including four windows across the two separate artificial seam sources.
|
|
Transcription errors, partial boundary phrases and overlap duplicates remain in
|
|
the evidence. Concept substring coverage is not a word-error score. Root
|
|
independently recomputed the exact recorded scorer output.
|
|
|
|
The complete report is 174868 bytes, SHA
|
|
`79ecb75d31c0193642c81862fd5476ad47adca518d4959e436bdd472cc281798`.
|
|
It uses the internal v3 CPU artifact and does not establish normal auto runtime
|
|
quality parity or GPU performance. Details and the retained comparison are in
|
|
[the evaluator](../tools/speech-eval/native-validation/README.md).
|
|
|
|
## Longer-source failure and release recovery
|
|
|
|
The normal-runtime timing regression retained all five original sources and
|
|
all eight queries, but the speech session returned a transport failure on the
|
|
first source and later files inherited the unavailable session. Raw Google
|
|
indexing completed with 79 chunks and no media failures. Recognized timing
|
|
scores remain ineligible; no historical 6/8 score is claimed for this run.
|
|
|
|
An isolated native reproduction using the **exact failed PCM hash**
|
|
`03ef72d2a4cb926b7d31fbaef757a6cb954a55d6d6961085ea6992728c8d9e37`
|
|
completed all four actual VAD intervals, valid recognition responses and exit
|
|
zero. The original transport failure is unexplained; no model or memory tuning
|
|
was introduced to hide it.
|
|
|
|
The release now restarts a verified session once at the next new source after
|
|
a terminal recoverable transport failure. The original failed source remains
|
|
reported; compatible donors return before restarting and raw search stays
|
|
usable. Malformed protocol/text and invalid token points never arm this reset.
|
|
Five focused service tests and native transport/lifecycle checks passed with
|
|
strict Clippy. Media and spoken passages remain visibly experimental.
|