Add local spoken passages with Google retrieval and isolated runtime setup

This commit is contained in:
karl committed 2026-10-13 20:18:44 +02:00
1 parent bc9459785d
commit dab703413e
55 files changed
+3633 -70

No files matched your search

+20
View File
@@ -224,3 +224,23 @@ synthetic corpus, exclude native UI and large-library/cold-disk claims, and do
not compare GPU performance. No timing-driven cache/architecture changes made.
Raw ignored verified report: `.searchi/qa/windows-mcp-013-timing-verified.json`,
SHA-256 `03c0a21aa35686077fca25e0b4fe7b7014d8f8aa7a07233af6518ac1b694db08`.
## Spoken sources in current development
When the shared index contains locally recognized speech, search results use
`semantic_speech` with a playback timestamp. MCP issues a bounded source handle
for that recognized passage. `read_source` verifies the current media bytes and
the exact issued passage, then returns recognized text, an end point and
`local_speech_recognition` provenance. Arbitrary timestamp or file-path reads
remain unavailable. Recognized text is untrusted evidence and can contain errors.
Google query configuration is unchanged: MCP reads the already prepared index;
it does not start speech recognition or need speech model paths. Headless
indexing can provision speech data with `searchi setup --speech` and opt into
recognition using the paired `embed --speech-worker PATH --speech-model-dir DIR`
options alongside the real Google configuration. The native speech worker and
models are separate from Google's worker and assets.
This integration is not present in the published Windows 0.1.5 or Linux 0.1.4
packages. Current checks and delivery gates are tracked in
[SPEECH-INTEGRATION.md](SPEECH-INTEGRATION.md).
+8
View File
@@ -40,6 +40,14 @@ Google EmbeddingGemma 2 remains the main text/image/audio/video search model.
preserved; no partial-pool scores were accepted. See SPEECH-DTW.md.
- [ ] Validate independent speech/languages and integrate accepted alignment with
model setup, packaging, cancellation, source guards and shared engine clients.
- [x] Implement isolated local recognition, independent raw/speech completion,
verified excerpts, desktop asset setup and CLI/MCP support; structural and
actual decoder-clock checks passed. See [integration](SPEECH-INTEGRATION.md).
- [x] Current CPU worker processed real English/German recordings from Unicode
paths and reused loaded recognition contexts; this is functional evidence,
not independent retrieval or word-timing accuracy.
- [ ] Run genuine shared-service Google lifecycle and frozen independent query
controls; stage and deliver the normal hardware-detecting runtime.
- [ ] Validate richer no-answer evidence on new independent positive/negative data; do not retune the failed universal cosine cutoff on old held-out results.
- [ ] Integrate accepted improvements into shared engine/UI/headless clients with verified setup, lifecycle and CPU performance evidence. Diagnostics alone do not raise the media quality rating.
- [x] Publish v0.1.0 Windows installer/checksum, verify public download bytes,
+65
View File
@@ -0,0 +1,65 @@
# Local spoken passages
Google EmbeddingGemma 2 remains the semantic model for text, images, audio and
video. An isolated local whisper.cpp worker adds recognized text to the same
Google index. It does not replace raw audio evidence or introduce a hosted
embedding backend.
The service covers an audio track with contiguous windows of at most thirty
seconds, overlapping by two seconds. Live Silero detection resets per window;
multilingual base recognition uses DTW token points. This differs from the
historical whole-source diagnostic in [SPEECH-DTW.md](SPEECH-DTW.md). Its 6/8
timing score cannot be carried over to this implementation.
Recognized passages retain source start/end points and exact recognized UTF-8.
A point span can have zero duration. The UI starts playback at the estimated
start and explains recognition uncertainty in the small match detail. A source
point is not a verified complete utterance boundary.
Raw Google media and recognized speech have independent completion policies.
Adding or retrying recognition reuses compatible raw vectors. Optional speech
failures preserve raw search evidence and report partial coverage. Excerpts
require current source bytes/revisions and the issued recognized passage;
same-stat overwrites invalidate them. CLI, UI and MCP use this shared behavior.
Desktop setup downloads and verifies approximately 149 MB of base/VAD model
data in addition to Google assets. Google query readiness stays independent.
The native worker is shipped in an isolated runtime directory. Normal packages
require both acceleration and CPU support; Windows VM execution is CPU only.
Automatic retries use source revision and speech policy, preventing generation
publication from creating an endless retry loop.
Each native call uses one admitted background CPU thread. Detailed indexing
metrics separate admission, decoding, VAD, model loading, ASR, transport and
fallback cost. A running native call cannot be preempted by a foreground query
on a one-thread budget. Raw media decoding and ANN work are outside this
inference budget.
## Evidence in this source change
- Shared lifecycle, passage bounds, partial failures, provenance and policy
tests passed. The complete workspace test command passed with genuine-model
tests explicitly ignored pending their dedicated run.
- Two genuine CPU FFmpeg tests passed: a two-second audio origin, an internal
timestamp gap, seeks and tails preserve the source timeline.
- Desktop Rust: 48 tests and strict Clippy passed. Frontend: 62 tests and
formatting passed, including a stable partial-speech retry key.
- The actual CLI speech setup verified/reused the pinned base and Silero cache.
- Existing atomic staging tests passed; Windows package tests reject missing,
changed, stale and internal CPU-only speech runtimes before replacing an
existing archive.
## Checks still required before delivery
- [ ] Current native worker English/German context reuse and Unicode paths.
- [ ] Actual shared-service ASR plus Google lifecycle integration.
- [ ] Frozen independent four-file/ten-query and separate seam evaluation.
- [ ] Normal hardware-detecting runtime staging and Windows package startup.
- [ ] Publish/install the verified Windows update.
- [ ] Native playback and graphical interaction, actual GPU execution, and
other OS runtime checks.
The independent fixture has no publisher phrase timestamps. It can measure
retrieval and negative controls, but cannot establish precise word-level timing
accuracy. Music relevance and general no-answer rejection remain separate open
quality work; diagnostic integration alone does not raise the media rating.