3.8 KiB
Actual shared-service speech validation
This internal evaluator uses the real SpeechWorker, shared service publication
and actual Google EmbeddingGemma 2. It consumes the unchanged frozen
fixtures/speech-validation-v1 sources and queries. No fixture reference text,
keywords or expected results enter recognition or indexed passages.
Run from the repository root after committing the evaluator/source snapshot and
obtaining the coordinator's exclusive CPU inference slot. The normal native
runtime is forced to CPU here; the shown cpu-v3 speech runtime is an internal
test artifact and must be verified by its owner before execution.
cargo test --manifest-path tools/speech-eval/native-validation/Cargo.toml --locked
python -m unittest discover -s tools/speech-eval -p test_score_native_validation.py
$env:SEARCHI_FORCE_CPU = '1'
cargo run --manifest-path tools/speech-eval/native-validation/Cargo.toml --locked -- fixtures/speech-validation-v1/fixture.json .searchi/speech-validation-v1-seams/prepared.json .searchi/models/embeddinggemma2 .searchi/runtime/auto/Release/searchi-gemma-worker.exe .searchi/speech-runtime/cpu-v3/runtime/searchi-speech-worker.exe .searchi/speech/models/ggml-base.bin .searchi/speech/models/ggml-silero-v6.2.0.bin .searchi/media-runtime/bin/ffmpeg.exe .searchi/media-runtime/bin/ffprobe.exe .searchi/qa/speech-service-validation-v1
python tools/speech-eval/score_native_validation.py .searchi/qa/speech-service-validation-v1/report.json fixtures/speech-validation-v1/fixture.json .searchi/qa/speech-service-validation-v1/scored.json
The output directory must not already exist. Individual native-call and query files, preflight, indexed state and case reports are immutable, with each file flushed before continuing. Failure writes an explicit incomplete artifact while preserving completed calls. Local databases/owned copied sources remain there for inspection. Model/runtime/source hashes, policy fingerprints and executing binary hash are recorded before inference. Freeze and rebuild shared sources before a scored run; on-disk source hashes alone cannot prove dependency compile provenance while another agent is editing.
The core case retains all four prepared files and all ten frozen queries.
Speech calls are matched against independently decoded source-window PCM hashes.
Accepted native SpeechTranscription values or returned errors are stored exactly
through the public client API. Native wire bytes and stderr are not exposed by
that API, so these artifacts are not claimed to capture raw transport output.
Published speech text/source hashes/times and byte-verified service excerpts are
retained, and foreground search uses the actual query role with measured traces.
The second case indexes both separate artificial seam sources using the same
service window policy. It checks source coverage and publication bounds; it does
not compare against continuously initialized full-source VAD.
The scorer independently verifies complete native PCM-window coverage, every frozen query, result-source identity and negative controls. It reports positive file recall/MRR, absent queries returning candidates, recognized negative-control passages and literal publisher-concept substring coverage separately. Concept matching is a diagnostic, not WER or an independent timed transcript.
There are no publisher phrase timestamps, so no strict word/sentence point quality score is produced. A timestamp merely inside the complete article is not precise localization success. Any excluded/failed source invalidates accepted comparison scoring, and hallucinations on music/silence remain measured failures. No provider, ranking weight, rejection threshold or production policy is chosen from this small experiment. The two readings share translated subject matter and the seam controls repeat real samples artificially.