180 lines
12 KiB
Markdown
180 lines
12 KiB
Markdown
# Actual shared-service speech validation
|
|
|
|
This internal evaluator uses the real `SpeechWorker`, shared service publication
|
|
and actual Google EmbeddingGemma 2. It consumes the unchanged frozen
|
|
`fixtures/speech-validation-v1` sources and queries. No fixture reference text,
|
|
keywords or expected results enter recognition or indexed passages.
|
|
|
|
Run from the repository root after committing the evaluator/source snapshot and
|
|
obtaining the coordinator's exclusive CPU inference slot. The normal native
|
|
runtime is forced to CPU here; the shown `cpu-v3` speech runtime is an internal
|
|
test artifact and must be verified by its owner before execution.
|
|
|
|
```powershell
|
|
cargo test --manifest-path tools/speech-eval/native-validation/Cargo.toml --locked
|
|
python -m unittest discover -s tools/speech-eval -p test_score_native_validation.py
|
|
$env:SEARCHI_FORCE_CPU = '1'
|
|
cargo run --manifest-path tools/speech-eval/native-validation/Cargo.toml --locked -- fixtures/speech-validation-v1/fixture.json .searchi/speech-validation-v1-seams/prepared.json .searchi/models/embeddinggemma2 .searchi/runtime/auto/Release/searchi-gemma-worker.exe .searchi/speech-runtime/cpu-v3/runtime/searchi-speech-worker.exe .searchi/speech/models/ggml-base.bin .searchi/speech/models/ggml-silero-v6.2.0.bin .searchi/media-runtime/bin/ffmpeg.exe .searchi/media-runtime/bin/ffprobe.exe .searchi/qa/speech-service-validation-v1
|
|
python tools/speech-eval/score_native_validation.py .searchi/qa/speech-service-validation-v1/report.json fixtures/speech-validation-v1/fixture.json .searchi/qa/speech-service-validation-v1/scored.json
|
|
```
|
|
|
|
The output directory must not already exist. Individual native-call and query
|
|
files, preflight, indexed state and case reports are immutable, with each file
|
|
flushed before continuing. Failure writes an explicit incomplete artifact while
|
|
preserving completed calls. Local databases/owned copied sources remain there
|
|
for inspection. Model/runtime/source hashes, policy fingerprints and executing
|
|
binary hash are recorded before inference. Freeze and rebuild shared sources
|
|
before a scored run; on-disk source hashes alone cannot prove dependency compile
|
|
provenance while another agent is editing.
|
|
|
|
The core case retains all four prepared files and all ten frozen queries.
|
|
Speech calls are matched against independently decoded source-window PCM hashes.
|
|
Accepted native `SpeechTranscription` values or returned errors are stored exactly
|
|
through the public client API. **Native wire bytes and stderr are not exposed by
|
|
that API**, so these artifacts are not claimed to capture raw transport output.
|
|
Published speech text/source hashes/times and byte-verified service excerpts are
|
|
retained, and foreground search uses the actual query role with measured traces.
|
|
The second case indexes both separate artificial seam sources using the same
|
|
service window policy. It checks source coverage and publication bounds; it does
|
|
not compare against continuously initialized full-source VAD.
|
|
|
|
The scorer independently verifies complete native PCM-window coverage, every
|
|
frozen query, result-source identity and negative controls. It reports positive
|
|
file recall/MRR, absent queries returning candidates, recognized negative-control
|
|
passages and literal publisher-concept substring coverage separately. Concept
|
|
matching is a diagnostic, not WER or an independent timed transcript.
|
|
|
|
There are no publisher phrase timestamps, so **no strict word/sentence point
|
|
quality score is produced**. A timestamp merely inside the complete article is
|
|
not precise localization success. Any excluded/failed source invalidates accepted
|
|
comparison scoring, and hallucinations on music/silence remain measured failures.
|
|
No provider, ranking weight, rejection threshold or production policy is chosen
|
|
from this small experiment. The two readings share translated subject matter and
|
|
the seam controls repeat real samples artificially.
|
|
|
|
## Completed v1 CPU execution
|
|
|
|
The frozen evaluator/source snapshot (`48f11de` / service `dab7034`, run at
|
|
`2f9190c`) completed once with the verified internal CPU v3 runtime. All four
|
|
core controls, ten queries, two seam sources and eight native source windows
|
|
were retained. Every PCM window matched the independently decoded expected
|
|
source; published recognized excerpts were byte-verified through the service.
|
|
Both cases have zero speech/indexing failures and eligible complete comparisons.
|
|
|
|
- Six positive queries retrieved a relevant file first: **6/6**, recall@5 **6/6**,
|
|
MRR **1.000**. Either translated recording is relevant under the frozen labels.
|
|
- Music and exact silence produced **zero accepted speech passages**.
|
|
- All four absent-topic queries still returned files: **4/4 no-answer failures**.
|
|
- English and German each retain all seven reference **substrings**. This is not
|
|
exact recognition quality: German contains `Artikel 1s` and
|
|
`Geisterbrüderlichkeit`, rather than the corresponding publisher phrasing.
|
|
- Seam windows preserve complete source coverage, bounds and citations, but
|
|
overlap text includes duplicate/partial or incorrect recognition, including
|
|
`Alemin`, `Einstiegleins` and `dignity and dignity`. These observations are
|
|
retained, not repaired or omitted. No precise point/localization score follows.
|
|
|
|
Core indexing took **127.61 seconds**, with **61.57 seconds** reported ASR.
|
|
Seam indexing took **163.37 seconds**, with **121.23 seconds** reported ASR.
|
|
Index timers include decoding, native work, Google and storage; native substage
|
|
times are nested, not additive with the full timer. Runtime/model verification
|
|
and query setup/warmup are separate in the summary. Normal Auto-runtime compilation
|
|
with one build job could overlap these shared Windows VM measurements; they are
|
|
not isolated hardware throughput. Eight logical CPUs and 16708012 KiB visible RAM
|
|
were exposed, but inference deliberately used one CPU thread. No normal Auto/GPU
|
|
quality or performance conclusion is inferred from this internal CPU artifact.
|
|
|
|
[Recomputed summary](evaluation-v1-summary.json) records every query outcome,
|
|
runtime/model/source hashes, cases and limitations. Immutable ignored artifacts:
|
|
|
|
| Artifact | Bytes | SHA-256 |
|
|
| --- | ---: | --- |
|
|
| `preflight.json` | 2075 | `63a0a615d37766b852c34473712156e950aeeb5a472a709bff6be00b7b26cb7a` |
|
|
| `report.json` | 174868 | `79ecb75d31c0193642c81862fd5476ad47adca518d4959e436bdd472cc281798` |
|
|
| `scored.json` | 5608 | `21d9bc84410cc63b96e2b575d99cd3c09834dfd9d2db2d0f356159df12e048e9` |
|
|
|
|
These files remain under `.searchi/qa/speech-service-validation-v1`, together
|
|
with every accepted native-call/query record and owned source/database. No
|
|
fixture, ranking parameter or provider was tuned from these results. Production
|
|
integration has meaningful real evidence here, while representative German
|
|
speech, precise publisher-timed localization, seam text handling, native UI,
|
|
broader media relevance and calibrated absence remain open.
|
|
|
|
## Frozen original media timing regression mode
|
|
|
|
`--media-v1-timing` reuses the same shared-service evaluator for the original
|
|
five complete sources: NASA `clip04`, piano `music01`, violin `music02`, bark
|
|
`sound01` and roaring effect `sound02`. All original source bytes and extensions
|
|
are preserved. This runs the actual live-window policy, not the older full-source
|
|
VAD decisions. It retains every raw media lane and every recognition output/error;
|
|
the 254-second piano's ten overlapping windows are not dropped as irrelevant.
|
|
|
|
The eight original publisher-timed speech queries/labels and fixture SHA remain
|
|
unchanged. The small additional timing policy pins the normal Auto-distribution
|
|
runtime, Google worker, decoders and existing `score.py` scoring-function hash.
|
|
Its SHA-256 is
|
|
`816b727f0447f2412b1fbe03a32ad928bdbe9c4f440bc0e7543e4b89c4dad3f0`.
|
|
Normal Auto compilation is distinct from this VM's forced CPU execution.
|
|
|
|
```powershell
|
|
cargo test --manifest-path tools/speech-eval/native-validation/Cargo.toml --locked -- --include-ignored
|
|
python -m unittest discover -s tools/speech-eval -p test_score_service_timing.py
|
|
$env:SEARCHI_FORCE_CPU = '1'
|
|
cargo run --manifest-path tools/speech-eval/native-validation/Cargo.toml --locked -- --media-v1-timing fixtures/media/fixture.json tools/speech-eval/native-validation/media-timing-policy.json C:/Users/karl/.searchi/models/embeddinggemma2 .searchi/runtime/auto/Release/searchi-gemma-worker.exe .searchi/s-a1/runtime/searchi-speech-worker.exe .searchi/speech/models/ggml-base.bin .searchi/speech/models/ggml-silero-v6.2.0.bin .searchi/media-runtime/bin/ffmpeg.exe .searchi/media-runtime/bin/ffprobe.exe .searchi/qa/speech-live-timing-v1
|
|
python tools/speech-eval/score_service_timing.py .searchi/qa/speech-live-timing-v1/report.json fixtures/media/fixture.json tools/speech-eval/native-validation/media-timing-policy.json .searchi/qa/speech-live-timing-v1/scored.json
|
|
```
|
|
|
|
The real-source preflight test verifies all five hashes and original extensions
|
|
without loading any model. Freeze/commit the evaluator policy before inference
|
|
and obtain the exclusive CPU slot. Original video/frame and music encoding can
|
|
take several minutes; this narrow timing regression does not change indexing
|
|
budgets, model/ranking policy or source files to make the run cheaper.
|
|
|
|
The adapter calls the existing publisher timing scorer's unchanged `score`
|
|
function. It uses only the actual hybrid **first result** and its byte-verified
|
|
recognized-speech excerpt's start/end. A raw audio/video first result remains an
|
|
explicit recognized-speech timing failure, with its original result retained;
|
|
no lower-ranked passage or label-selected timestamp replaces it. Failed or
|
|
missing source windows prohibit numeric accepted timing scores. Historical
|
|
full-source gated DTW **6/8** is context, never an inherited result or an isolated
|
|
causal comparison with this end-to-end hybrid run. No new timing score exists
|
|
until this exact mode executes and all checks finish.
|
|
|
|
The first attempt at `.searchi/qa/speech-live-timing-v1` failed during source
|
|
preflight, before any model load or inference. Its immutable `failure.json`
|
|
SHA-256 is `9ba49fd5cdb69e57c267f579818f226949b02eafbc0e592161fad7593f00c6aa`.
|
|
The bark recording lasts 2.038594 seconds: its rounded 2039 ms window implies
|
|
32624 samples, while aligned decoding correctly retains 32618 real samples.
|
|
The six-sample difference is 0.375 ms. The evaluator now accepts the existing
|
|
one-millisecond coverage bound (16 samples), without padding, trimming or
|
|
changing source bytes, labels, ranking, or DTW points. Its regression test
|
|
accepts that real tail and rejects a 17-sample discrepancy and invalid bounds.
|
|
Seven Rust tests, strict all-target Clippy and five timing-adapter tests passed.
|
|
A retry must use a fresh output directory after the corrected evaluator is
|
|
frozen and the CPU inference slot is granted again; the failed attempt remains
|
|
preserved and has no quality score.
|
|
|
|
The corrected frozen attempt (commit `f15ee07`, session 39812) completed on CPU
|
|
at `.searchi/qa/speech-live-timing-v1-coverage`, but is **not timing-eligible**.
|
|
All five source checks passed; raw indexing produced 79 chunks (16 audio and
|
|
63 image calls), with no media failures, in 377691.6842 ms. The first speech
|
|
request for the video's initial 30-second window failed with
|
|
`Speech worker transport exited or timed out` after 2382.0169 ms. The four
|
|
remaining sources reported `Unavailable speech session or unbounded PCM`.
|
|
Five failed calls were recorded against 16 expected windows; zero recognized
|
|
speech chunks or citations were published. The public client discards stderr,
|
|
so these artifacts cannot identify the native failure cause. The request
|
|
deadline is 600 seconds, longer than the observed first-call failure.
|
|
|
|
All eight first files were `clip04`, but their actual origins were raw audio or
|
|
video, not recognized speech. The unchanged scorer therefore records
|
|
`comparison_eligible: false`, `strict_point_score: null` and
|
|
`correct_first_file: null`; no historical score carries over.
|
|
Immutable report SHA-256:
|
|
`7261223ba479ff1c319ded187fcd6e7cf37d351ec9d62437e26b63017d1bccc2`;
|
|
scored artifact SHA-256:
|
|
`361d5fbaf917363bfedd9c3ba2ea368d1714cf28f1bc34544a9ed324761eb4db`;
|
|
preflight SHA-256:
|
|
`97c603b3078831ce0fcf19f0ebb9e15160ee2363b488946c992e881d5509ce6b`.
|
|
This failed speech lifecycle requires investigation before any live-window
|
|
timing claim; no policy, labels or sources were changed after this result.
|