Add local spoken passages with Google retrieval and isolated runtime setup

This commit is contained in:
karl committed 2026-10-13 20:18:44 +02:00
1 parent bc9459785d
commit dab703413e
55 files changed
+3633 -70

No files matched your search

+65
View File
@@ -0,0 +1,65 @@
# Local spoken passages
Google EmbeddingGemma 2 remains the semantic model for text, images, audio and
video. An isolated local whisper.cpp worker adds recognized text to the same
Google index. It does not replace raw audio evidence or introduce a hosted
embedding backend.
The service covers an audio track with contiguous windows of at most thirty
seconds, overlapping by two seconds. Live Silero detection resets per window;
multilingual base recognition uses DTW token points. This differs from the
historical whole-source diagnostic in [SPEECH-DTW.md](SPEECH-DTW.md). Its 6/8
timing score cannot be carried over to this implementation.
Recognized passages retain source start/end points and exact recognized UTF-8.
A point span can have zero duration. The UI starts playback at the estimated
start and explains recognition uncertainty in the small match detail. A source
point is not a verified complete utterance boundary.
Raw Google media and recognized speech have independent completion policies.
Adding or retrying recognition reuses compatible raw vectors. Optional speech
failures preserve raw search evidence and report partial coverage. Excerpts
require current source bytes/revisions and the issued recognized passage;
same-stat overwrites invalidate them. CLI, UI and MCP use this shared behavior.
Desktop setup downloads and verifies approximately 149 MB of base/VAD model
data in addition to Google assets. Google query readiness stays independent.
The native worker is shipped in an isolated runtime directory. Normal packages
require both acceleration and CPU support; Windows VM execution is CPU only.
Automatic retries use source revision and speech policy, preventing generation
publication from creating an endless retry loop.
Each native call uses one admitted background CPU thread. Detailed indexing
metrics separate admission, decoding, VAD, model loading, ASR, transport and
fallback cost. A running native call cannot be preempted by a foreground query
on a one-thread budget. Raw media decoding and ANN work are outside this
inference budget.
## Evidence in this source change
- Shared lifecycle, passage bounds, partial failures, provenance and policy
tests passed. The complete workspace test command passed with genuine-model
tests explicitly ignored pending their dedicated run.
- Two genuine CPU FFmpeg tests passed: a two-second audio origin, an internal
timestamp gap, seeks and tails preserve the source timeline.
- Desktop Rust: 48 tests and strict Clippy passed. Frontend: 62 tests and
formatting passed, including a stable partial-speech retry key.
- The actual CLI speech setup verified/reused the pinned base and Silero cache.
- Existing atomic staging tests passed; Windows package tests reject missing,
changed, stale and internal CPU-only speech runtimes before replacing an
existing archive.
## Checks still required before delivery
- [ ] Current native worker English/German context reuse and Unicode paths.
- [ ] Actual shared-service ASR plus Google lifecycle integration.
- [ ] Frozen independent four-file/ten-query and separate seam evaluation.
- [ ] Normal hardware-detecting runtime staging and Windows package startup.
- [ ] Publish/install the verified Windows update.
- [ ] Native playback and graphical interaction, actual GPU execution, and
other OS runtime checks.
The independent fixture has no publisher phrase timestamps. It can measure
retrieval and negative controls, but cannot establish precise word-level timing
accuracy. Music relevance and general no-answer rejection remain separate open
quality work; diagnostic integration alone does not raise the media rating.