Add local spoken passages with Google retrieval and isolated runtime setup
This commit is contained in:
1 parent
bc9459785d
commit
dab703413e
55 files changed
+3633
-70
No files matched your search
@@ -224,3 +224,23 @@ synthetic corpus, exclude native UI and large-library/cold-disk claims, and do
|
||||
not compare GPU performance. No timing-driven cache/architecture changes made.
|
||||
Raw ignored verified report: `.searchi/qa/windows-mcp-013-timing-verified.json`,
|
||||
SHA-256 `03c0a21aa35686077fca25e0b4fe7b7014d8f8aa7a07233af6518ac1b694db08`.
|
||||
|
||||
## Spoken sources in current development
|
||||
|
||||
When the shared index contains locally recognized speech, search results use
|
||||
`semantic_speech` with a playback timestamp. MCP issues a bounded source handle
|
||||
for that recognized passage. `read_source` verifies the current media bytes and
|
||||
the exact issued passage, then returns recognized text, an end point and
|
||||
`local_speech_recognition` provenance. Arbitrary timestamp or file-path reads
|
||||
remain unavailable. Recognized text is untrusted evidence and can contain errors.
|
||||
|
||||
Google query configuration is unchanged: MCP reads the already prepared index;
|
||||
it does not start speech recognition or need speech model paths. Headless
|
||||
indexing can provision speech data with `searchi setup --speech` and opt into
|
||||
recognition using the paired `embed --speech-worker PATH --speech-model-dir DIR`
|
||||
options alongside the real Google configuration. The native speech worker and
|
||||
models are separate from Google's worker and assets.
|
||||
|
||||
This integration is not present in the published Windows 0.1.5 or Linux 0.1.4
|
||||
packages. Current checks and delivery gates are tracked in
|
||||
[SPEECH-INTEGRATION.md](SPEECH-INTEGRATION.md).
|
||||
@@ -40,6 +40,14 @@ Google EmbeddingGemma 2 remains the main text/image/audio/video search model.
|
||||
preserved; no partial-pool scores were accepted. See SPEECH-DTW.md.
|
||||
- [ ] Validate independent speech/languages and integrate accepted alignment with
|
||||
model setup, packaging, cancellation, source guards and shared engine clients.
|
||||
- [x] Implement isolated local recognition, independent raw/speech completion,
|
||||
verified excerpts, desktop asset setup and CLI/MCP support; structural and
|
||||
actual decoder-clock checks passed. See [integration](SPEECH-INTEGRATION.md).
|
||||
- [x] Current CPU worker processed real English/German recordings from Unicode
|
||||
paths and reused loaded recognition contexts; this is functional evidence,
|
||||
not independent retrieval or word-timing accuracy.
|
||||
- [ ] Run genuine shared-service Google lifecycle and frozen independent query
|
||||
controls; stage and deliver the normal hardware-detecting runtime.
|
||||
- [ ] Validate richer no-answer evidence on new independent positive/negative data; do not retune the failed universal cosine cutoff on old held-out results.
|
||||
- [ ] Integrate accepted improvements into shared engine/UI/headless clients with verified setup, lifecycle and CPU performance evidence. Diagnostics alone do not raise the media quality rating.
|
||||
- [x] Publish v0.1.0 Windows installer/checksum, verify public download bytes,
|
||||
|
||||
@@ -0,0 +1,65 @@
|
||||
# Local spoken passages
|
||||
|
||||
Google EmbeddingGemma 2 remains the semantic model for text, images, audio and
|
||||
video. An isolated local whisper.cpp worker adds recognized text to the same
|
||||
Google index. It does not replace raw audio evidence or introduce a hosted
|
||||
embedding backend.
|
||||
|
||||
The service covers an audio track with contiguous windows of at most thirty
|
||||
seconds, overlapping by two seconds. Live Silero detection resets per window;
|
||||
multilingual base recognition uses DTW token points. This differs from the
|
||||
historical whole-source diagnostic in [SPEECH-DTW.md](SPEECH-DTW.md). Its 6/8
|
||||
timing score cannot be carried over to this implementation.
|
||||
|
||||
Recognized passages retain source start/end points and exact recognized UTF-8.
|
||||
A point span can have zero duration. The UI starts playback at the estimated
|
||||
start and explains recognition uncertainty in the small match detail. A source
|
||||
point is not a verified complete utterance boundary.
|
||||
|
||||
Raw Google media and recognized speech have independent completion policies.
|
||||
Adding or retrying recognition reuses compatible raw vectors. Optional speech
|
||||
failures preserve raw search evidence and report partial coverage. Excerpts
|
||||
require current source bytes/revisions and the issued recognized passage;
|
||||
same-stat overwrites invalidate them. CLI, UI and MCP use this shared behavior.
|
||||
|
||||
Desktop setup downloads and verifies approximately 149 MB of base/VAD model
|
||||
data in addition to Google assets. Google query readiness stays independent.
|
||||
The native worker is shipped in an isolated runtime directory. Normal packages
|
||||
require both acceleration and CPU support; Windows VM execution is CPU only.
|
||||
Automatic retries use source revision and speech policy, preventing generation
|
||||
publication from creating an endless retry loop.
|
||||
|
||||
Each native call uses one admitted background CPU thread. Detailed indexing
|
||||
metrics separate admission, decoding, VAD, model loading, ASR, transport and
|
||||
fallback cost. A running native call cannot be preempted by a foreground query
|
||||
on a one-thread budget. Raw media decoding and ANN work are outside this
|
||||
inference budget.
|
||||
|
||||
## Evidence in this source change
|
||||
|
||||
- Shared lifecycle, passage bounds, partial failures, provenance and policy
|
||||
tests passed. The complete workspace test command passed with genuine-model
|
||||
tests explicitly ignored pending their dedicated run.
|
||||
- Two genuine CPU FFmpeg tests passed: a two-second audio origin, an internal
|
||||
timestamp gap, seeks and tails preserve the source timeline.
|
||||
- Desktop Rust: 48 tests and strict Clippy passed. Frontend: 62 tests and
|
||||
formatting passed, including a stable partial-speech retry key.
|
||||
- The actual CLI speech setup verified/reused the pinned base and Silero cache.
|
||||
- Existing atomic staging tests passed; Windows package tests reject missing,
|
||||
changed, stale and internal CPU-only speech runtimes before replacing an
|
||||
existing archive.
|
||||
|
||||
## Checks still required before delivery
|
||||
|
||||
- [ ] Current native worker English/German context reuse and Unicode paths.
|
||||
- [ ] Actual shared-service ASR plus Google lifecycle integration.
|
||||
- [ ] Frozen independent four-file/ten-query and separate seam evaluation.
|
||||
- [ ] Normal hardware-detecting runtime staging and Windows package startup.
|
||||
- [ ] Publish/install the verified Windows update.
|
||||
- [ ] Native playback and graphical interaction, actual GPU execution, and
|
||||
other OS runtime checks.
|
||||
|
||||
The independent fixture has no publisher phrase timestamps. It can measure
|
||||
retrieval and negative controls, but cannot establish precise word-level timing
|
||||
accuracy. Music relevance and general no-answer rejection remain separate open
|
||||
quality work; diagnostic integration alone does not raise the media rating.
|
||||
Reference in new issue
Block a user