This internal evaluator uses the real SpeechWorker, shared service publication and actual Google EmbeddingGemma 2. It consumes the unchanged frozen fixtures/speech-validation-v1 sources and queries. No fixture reference text, keywords or expected results enter recognition or indexed passages.

Run from the repository root after committing the evaluator/source snapshot and obtaining the coordinator's exclusive CPU inference slot. The normal native runtime is forced to CPU here; the shown cpu-v3 speech runtime is an internal test artifact and must be verified by its owner before execution.

cargo test --manifest-path tools/speech-eval/native-validation/Cargo.toml --locked
python -m unittest discover -s tools/speech-eval -p test_score_native_validation.py
$env:SEARCHI_FORCE_CPU = '1'
cargo run --manifest-path tools/speech-eval/native-validation/Cargo.toml --locked -- fixtures/speech-validation-v1/fixture.json .searchi/speech-validation-v1-seams/prepared.json .searchi/models/embeddinggemma2 .searchi/runtime/auto/Release/searchi-gemma-worker.exe .searchi/speech-runtime/cpu-v3/runtime/searchi-speech-worker.exe .searchi/speech/models/ggml-base.bin .searchi/speech/models/ggml-silero-v6.2.0.bin .searchi/media-runtime/bin/ffmpeg.exe .searchi/media-runtime/bin/ffprobe.exe .searchi/qa/speech-service-validation-v1
python tools/speech-eval/score_native_validation.py .searchi/qa/speech-service-validation-v1/report.json fixtures/speech-validation-v1/fixture.json .searchi/qa/speech-service-validation-v1/scored.json

The output directory must not already exist. Individual native-call and query files, preflight, indexed state and case reports are immutable, with each file flushed before continuing. Failure writes an explicit incomplete artifact while preserving completed calls. Local databases/owned copied sources remain there for inspection. Model/runtime/source hashes, policy fingerprints and executing binary hash are recorded before inference. Freeze and rebuild shared sources before a scored run; on-disk source hashes alone cannot prove dependency compile provenance while another agent is editing.

The core case retains all four prepared files and all ten frozen queries. Speech calls are matched against independently decoded source-window PCM hashes. Accepted native SpeechTranscription values or returned errors are stored exactly through the public client API. Native wire bytes and stderr are not exposed by that API, so these artifacts are not claimed to capture raw transport output. Published speech text/source hashes/times and byte-verified service excerpts are retained, and foreground search uses the actual query role with measured traces. The second case indexes both separate artificial seam sources using the same service window policy. It checks source coverage and publication bounds; it does not compare against continuously initialized full-source VAD.

The scorer independently verifies complete native PCM-window coverage, every frozen query, result-source identity and negative controls. It reports positive file recall/MRR, absent queries returning candidates, recognized negative-control passages and literal publisher-concept substring coverage separately. Concept matching is a diagnostic, not WER or an independent timed transcript.

There are no publisher phrase timestamps, so no strict word/sentence point quality score is produced. A timestamp merely inside the complete article is not precise localization success. Any excluded/failed source invalidates accepted comparison scoring, and hallucinations on music/silence remain measured failures. No provider, ranking weight, rejection threshold or production policy is chosen from this small experiment. The two readings share translated subject matter and the seam controls repeat real samples artificially.

Completed v1 CPU execution

The frozen evaluator/source snapshot (48f11de / service dab7034, run at 2f9190c) completed once with the verified internal CPU v3 runtime. All four core controls, ten queries, two seam sources and eight native source windows were retained. Every PCM window matched the independently decoded expected source; published recognized excerpts were byte-verified through the service. Both cases have zero speech/indexing failures and eligible complete comparisons.

Six positive queries retrieved a relevant file first: 6/6, recall@5 6/6, MRR 1.000. Either translated recording is relevant under the frozen labels.
Music and exact silence produced zero accepted speech passages.
All four absent-topic queries still returned files: 4/4 no-answer failures.
English and German each retain all seven reference substrings. This is not exact recognition quality: German contains Artikel 1s and Geisterbrüderlichkeit, rather than the corresponding publisher phrasing.
Seam windows preserve complete source coverage, bounds and citations, but overlap text includes duplicate/partial or incorrect recognition, including Alemin, Einstiegleins and dignity and dignity. These observations are retained, not repaired or omitted. No precise point/localization score follows.

Core indexing took 127.61 seconds, with 61.57 seconds reported ASR. Seam indexing took 163.37 seconds, with 121.23 seconds reported ASR. Index timers include decoding, native work, Google and storage; native substage times are nested, not additive with the full timer. Runtime/model verification and query setup/warmup are separate in the summary. Normal Auto-runtime compilation with one build job could overlap these shared Windows VM measurements; they are not isolated hardware throughput. Eight logical CPUs and 16708012 KiB visible RAM were exposed, but inference deliberately used one CPU thread. No normal Auto/GPU quality or performance conclusion is inferred from this internal CPU artifact.

Recomputed summary records every query outcome, runtime/model/source hashes, cases and limitations. Immutable ignored artifacts:

Artifact	Bytes	SHA-256
`preflight.json`	2075	`63a0a615d37766b852c34473712156e950aeeb5a472a709bff6be00b7b26cb7a`
`report.json`	174868	`79ecb75d31c0193642c81862fd5476ad47adca518d4959e436bdd472cc281798`
`scored.json`	5608	`21d9bc84410cc63b96e2b575d99cd3c09834dfd9d2db2d0f356159df12e048e9`

These files remain under .searchi/qa/speech-service-validation-v1, together with every accepted native-call/query record and owned source/database. No fixture, ranking parameter or provider was tuned from these results. Production integration has meaningful real evidence here, while representative German speech, precise publisher-timed localization, seam text handling, native UI, broader media relevance and calibrated absence remain open.

Frozen original media timing regression mode

--media-v1-timing reuses the same shared-service evaluator for the original five complete sources: NASA clip04, piano music01, violin music02, bark sound01 and roaring effect sound02. All original source bytes and extensions are preserved. This runs the actual live-window policy, not the older full-source VAD decisions. It retains every raw media lane and every recognition output/error; the 254-second piano's ten overlapping windows are not dropped as irrelevant.

The eight original publisher-timed speech queries/labels and fixture SHA remain unchanged. The small additional timing policy pins the normal Auto-distribution runtime, Google worker, decoders and existing score.py scoring-function hash. Its SHA-256 is 816b727f0447f2412b1fbe03a32ad928bdbe9c4f440bc0e7543e4b89c4dad3f0. Normal Auto compilation is distinct from this VM's forced CPU execution.

cargo test --manifest-path tools/speech-eval/native-validation/Cargo.toml --locked -- --include-ignored
python -m unittest discover -s tools/speech-eval -p test_score_service_timing.py
$env:SEARCHI_FORCE_CPU = '1'
cargo run --manifest-path tools/speech-eval/native-validation/Cargo.toml --locked -- --media-v1-timing fixtures/media/fixture.json tools/speech-eval/native-validation/media-timing-policy.json C:/Users/karl/.searchi/models/embeddinggemma2 .searchi/runtime/auto/Release/searchi-gemma-worker.exe .searchi/s-a1/runtime/searchi-speech-worker.exe .searchi/speech/models/ggml-base.bin .searchi/speech/models/ggml-silero-v6.2.0.bin .searchi/media-runtime/bin/ffmpeg.exe .searchi/media-runtime/bin/ffprobe.exe .searchi/qa/speech-live-timing-v1
python tools/speech-eval/score_service_timing.py .searchi/qa/speech-live-timing-v1/report.json fixtures/media/fixture.json tools/speech-eval/native-validation/media-timing-policy.json .searchi/qa/speech-live-timing-v1/scored.json

The real-source preflight test verifies all five hashes and original extensions without loading any model. Freeze/commit the evaluator policy before inference and obtain the exclusive CPU slot. Original video/frame and music encoding can take several minutes; this narrow timing regression does not change indexing budgets, model/ranking policy or source files to make the run cheaper.

The adapter calls the existing publisher timing scorer's unchanged score function. It uses only the actual hybrid first result and its byte-verified recognized-speech excerpt's start/end. A raw audio/video first result remains an explicit recognized-speech timing failure, with its original result retained; no lower-ranked passage or label-selected timestamp replaces it. Failed or missing source windows prohibit numeric accepted timing scores. Historical full-source gated DTW 6/8 is context, never an inherited result or an isolated causal comparison with this end-to-end hybrid run. No new timing score exists until this exact mode executes and all checks finish.

The first attempt at .searchi/qa/speech-live-timing-v1 failed during source preflight, before any model load or inference. Its immutable failure.json SHA-256 is 9ba49fd5cdb69e57c267f579818f226949b02eafbc0e592161fad7593f00c6aa. The bark recording lasts 2.038594 seconds: its rounded 2039 ms window implies 32624 samples, while aligned decoding correctly retains 32618 real samples. The six-sample difference is 0.375 ms. The evaluator now accepts the existing one-millisecond coverage bound (16 samples), without padding, trimming or changing source bytes, labels, ranking, or DTW points. Its regression test accepts that real tail and rejects a 17-sample discrepancy and invalid bounds. Seven Rust tests, strict all-target Clippy and five timing-adapter tests passed. A retry must use a fresh output directory after the corrected evaluator is frozen and the CPU inference slot is granted again; the failed attempt remains preserved and has no quality score.

The corrected frozen attempt (commit f15ee07, session 39812) completed on CPU at .searchi/qa/speech-live-timing-v1-coverage, but is not timing-eligible. All five source checks passed; raw indexing produced 79 chunks (16 audio and 63 image calls), with no media failures, in 377691.6842 ms. The first speech request for the video's initial 30-second window failed with Speech worker transport exited or timed out after 2382.0169 ms. The four remaining sources reported Unavailable speech session or unbounded PCM. Five failed calls were recorded against 16 expected windows; zero recognized speech chunks or citations were published. The public client discards stderr, so these artifacts cannot identify the native failure cause. The request deadline is 600 seconds, longer than the observed first-call failure.

All eight first files were clip04, but their actual origins were raw audio or video, not recognized speech. The unchanged scorer therefore records comparison_eligible: false, strict_point_score: null and correct_first_file: null; no historical score carries over. Immutable report SHA-256: 7261223ba479ff1c319ded187fcd6e7cf37d351ec9d62437e26b63017d1bccc2; scored artifact SHA-256: 361d5fbaf917363bfedd9c3ba2ea368d1714cf28f1bc34544a9ed324761eb4db; preflight SHA-256: 97c603b3078831ce0fcf19f0ebb9e15160ee2363b488946c992e881d5509ce6b. This failed speech lifecycle requires investigation before any live-window timing claim; no policy, labels or sources were changed after this result.