# Actual shared-service speech validation

This internal evaluator uses the real `SpeechWorker`, shared service publication
and actual Google EmbeddingGemma 2. It consumes the unchanged frozen
`fixtures/speech-validation-v1` sources and queries. No fixture reference text,
keywords or expected results enter recognition or indexed passages.

Run from the repository root after committing the evaluator/source snapshot and
obtaining the coordinator's exclusive CPU inference slot. The normal native
runtime is forced to CPU here; the shown `cpu-v3` speech runtime is an internal
test artifact and must be verified by its owner before execution.

```powershell
cargo test --manifest-path tools/speech-eval/native-validation/Cargo.toml --locked
python -m unittest discover -s tools/speech-eval -p test_score_native_validation.py
$env:SEARCHI_FORCE_CPU = '1'
cargo run --manifest-path tools/speech-eval/native-validation/Cargo.toml --locked -- fixtures/speech-validation-v1/fixture.json .searchi/speech-validation-v1-seams/prepared.json .searchi/models/embeddinggemma2 .searchi/runtime/auto/Release/searchi-gemma-worker.exe .searchi/speech-runtime/cpu-v3/runtime/searchi-speech-worker.exe .searchi/speech/models/ggml-base.bin .searchi/speech/models/ggml-silero-v6.2.0.bin .searchi/media-runtime/bin/ffmpeg.exe .searchi/media-runtime/bin/ffprobe.exe .searchi/qa/speech-service-validation-v1
python tools/speech-eval/score_native_validation.py .searchi/qa/speech-service-validation-v1/report.json fixtures/speech-validation-v1/fixture.json .searchi/qa/speech-service-validation-v1/scored.json
```

The output directory must not already exist. Individual native-call and query
files, preflight, indexed state and case reports are immutable, with each file
flushed before continuing. Failure writes an explicit incomplete artifact while
preserving completed calls. Local databases/owned copied sources remain there
for inspection. Model/runtime/source hashes, policy fingerprints and executing
binary hash are recorded before inference. Freeze and rebuild shared sources
before a scored run; on-disk source hashes alone cannot prove dependency compile
provenance while another agent is editing.

The core case retains all four prepared files and all ten frozen queries.
Speech calls are matched against independently decoded source-window PCM hashes.
Accepted native `SpeechTranscription` values or returned errors are stored exactly
through the public client API. **Native wire bytes and stderr are not exposed by
that API**, so these artifacts are not claimed to capture raw transport output.
Published speech text/source hashes/times and byte-verified service excerpts are
retained, and foreground search uses the actual query role with measured traces.
The second case indexes both separate artificial seam sources using the same
service window policy. It checks source coverage and publication bounds; it does
not compare against continuously initialized full-source VAD.

The scorer independently verifies complete native PCM-window coverage, every
frozen query, result-source identity and negative controls. It reports positive
file recall/MRR, absent queries returning candidates, recognized negative-control
passages and literal publisher-concept substring coverage separately. Concept
matching is a diagnostic, not WER or an independent timed transcript.

There are no publisher phrase timestamps, so **no strict word/sentence point
quality score is produced**. A timestamp merely inside the complete article is
not precise localization success. Any excluded/failed source invalidates accepted
comparison scoring, and hallucinations on music/silence remain measured failures.
No provider, ranking weight, rejection threshold or production policy is chosen
from this small experiment. The two readings share translated subject matter and
the seam controls repeat real samples artificially.

## Completed v1 CPU execution

The frozen evaluator/source snapshot (`48f11de` / service `dab7034`, run at
`2f9190c`) completed once with the verified internal CPU v3 runtime. All four
core controls, ten queries, two seam sources and eight native source windows
were retained. Every PCM window matched the independently decoded expected
source; published recognized excerpts were byte-verified through the service.
Both cases have zero speech/indexing failures and eligible complete comparisons.

- Six positive queries retrieved a relevant file first: **6/6**, recall@5 **6/6**,
  MRR **1.000**. Either translated recording is relevant under the frozen labels.
- Music and exact silence produced **zero accepted speech passages**.
- All four absent-topic queries still returned files: **4/4 no-answer failures**.
- English and German each retain all seven reference **substrings**. This is not
  exact recognition quality: German contains `Artikel 1s` and
  `Geisterbrüderlichkeit`, rather than the corresponding publisher phrasing.
- Seam windows preserve complete source coverage, bounds and citations, but
  overlap text includes duplicate/partial or incorrect recognition, including
  `Alemin`, `Einstiegleins` and `dignity and dignity`. These observations are
  retained, not repaired or omitted. No precise point/localization score follows.

Core indexing took **127.61 seconds**, with **61.57 seconds** reported ASR.
Seam indexing took **163.37 seconds**, with **121.23 seconds** reported ASR.
Index timers include decoding, native work, Google and storage; native substage
times are nested, not additive with the full timer. Runtime/model verification
and query setup/warmup are separate in the summary. Normal Auto-runtime compilation
with one build job could overlap these shared Windows VM measurements; they are
not isolated hardware throughput. Eight logical CPUs and 16708012 KiB visible RAM
were exposed, but inference deliberately used one CPU thread. No normal Auto/GPU
quality or performance conclusion is inferred from this internal CPU artifact.

[Recomputed summary](evaluation-v1-summary.json) records every query outcome,
runtime/model/source hashes, cases and limitations. Immutable ignored artifacts:

| Artifact | Bytes | SHA-256 |
| --- | ---: | --- |
| `preflight.json` | 2075 | `63a0a615d37766b852c34473712156e950aeeb5a472a709bff6be00b7b26cb7a` |
| `report.json` | 174868 | `79ecb75d31c0193642c81862fd5476ad47adca518d4959e436bdd472cc281798` |
| `scored.json` | 5608 | `21d9bc84410cc63b96e2b575d99cd3c09834dfd9d2db2d0f356159df12e048e9` |

These files remain under `.searchi/qa/speech-service-validation-v1`, together
with every accepted native-call/query record and owned source/database. No
fixture, ranking parameter or provider was tuned from these results. Production
integration has meaningful real evidence here, while representative German
speech, precise publisher-timed localization, seam text handling, native UI,
broader media relevance and calibrated absence remain open.

## Frozen original media timing regression mode

`--media-v1-timing` reuses the same shared-service evaluator for the original
five complete sources: NASA `clip04`, piano `music01`, violin `music02`, bark
`sound01` and roaring effect `sound02`. All original source bytes and extensions
are preserved. This runs the actual live-window policy, not the older full-source
VAD decisions. It retains every raw media lane and every recognition output/error;
the 254-second piano's ten overlapping windows are not dropped as irrelevant.

The eight original publisher-timed speech queries/labels and fixture SHA remain
unchanged. The small additional timing policy pins the normal Auto-distribution
runtime, Google worker, decoders and existing `score.py` scoring-function hash.
Its SHA-256 is
`816b727f0447f2412b1fbe03a32ad928bdbe9c4f440bc0e7543e4b89c4dad3f0`.
Normal Auto compilation is distinct from this VM's forced CPU execution.

```powershell
cargo test --manifest-path tools/speech-eval/native-validation/Cargo.toml --locked -- --include-ignored
python -m unittest discover -s tools/speech-eval -p test_score_service_timing.py
$env:SEARCHI_FORCE_CPU = '1'
cargo run --manifest-path tools/speech-eval/native-validation/Cargo.toml --locked -- --media-v1-timing fixtures/media/fixture.json tools/speech-eval/native-validation/media-timing-policy.json C:/Users/karl/.searchi/models/embeddinggemma2 .searchi/runtime/auto/Release/searchi-gemma-worker.exe .searchi/s-a1/runtime/searchi-speech-worker.exe .searchi/speech/models/ggml-base.bin .searchi/speech/models/ggml-silero-v6.2.0.bin .searchi/media-runtime/bin/ffmpeg.exe .searchi/media-runtime/bin/ffprobe.exe .searchi/qa/speech-live-timing-v1
python tools/speech-eval/score_service_timing.py .searchi/qa/speech-live-timing-v1/report.json fixtures/media/fixture.json tools/speech-eval/native-validation/media-timing-policy.json .searchi/qa/speech-live-timing-v1/scored.json
```

The real-source preflight test verifies all five hashes and original extensions
without loading any model. Freeze/commit the evaluator policy before inference
and obtain the exclusive CPU slot. Original video/frame and music encoding can
take several minutes; this narrow timing regression does not change indexing
budgets, model/ranking policy or source files to make the run cheaper.

The adapter calls the existing publisher timing scorer's unchanged `score`
function. It uses only the actual hybrid **first result** and its byte-verified
recognized-speech excerpt's start/end. A raw audio/video first result remains an
explicit recognized-speech timing failure, with its original result retained;
no lower-ranked passage or label-selected timestamp replaces it. Failed or
missing source windows prohibit numeric accepted timing scores. Historical
full-source gated DTW **6/8** is context, never an inherited result or an isolated
causal comparison with this end-to-end hybrid run. No new timing score exists
until this exact mode executes and all checks finish.

The first attempt at `.searchi/qa/speech-live-timing-v1` failed during source
preflight, before any model load or inference. Its immutable `failure.json`
SHA-256 is `9ba49fd5cdb69e57c267f579818f226949b02eafbc0e592161fad7593f00c6aa`.
The bark recording lasts 2.038594 seconds: its rounded 2039 ms window implies
32624 samples, while aligned decoding correctly retains 32618 real samples.
The six-sample difference is 0.375 ms. The evaluator now accepts the existing
one-millisecond coverage bound (16 samples), without padding, trimming or
changing source bytes, labels, ranking, or DTW points. Its regression test
accepts that real tail and rejects a 17-sample discrepancy and invalid bounds.
Seven Rust tests, strict all-target Clippy and five timing-adapter tests passed.
A retry must use a fresh output directory after the corrected evaluator is
frozen and the CPU inference slot is granted again; the failed attempt remains
preserved and has no quality score.

The corrected frozen attempt (commit `f15ee07`, session 39812) completed on CPU
at `.searchi/qa/speech-live-timing-v1-coverage`, but is **not timing-eligible**.
All five source checks passed; raw indexing produced 79 chunks (16 audio and
63 image calls), with no media failures, in 377691.6842 ms. The first speech
request for the video's initial 30-second window failed with
`Speech worker transport exited or timed out` after 2382.0169 ms. The four
remaining sources reported `Unavailable speech session or unbounded PCM`.
Five failed calls were recorded against 16 expected windows; zero recognized
speech chunks or citations were published. The public client discards stderr,
so these artifacts cannot identify the native failure cause. The request
deadline is 600 seconds, longer than the observed first-call failure.

All eight first files were `clip04`, but their actual origins were raw audio or
video, not recognized speech. The unchanged scorer therefore records
`comparison_eligible: false`, `strict_point_score: null` and
`correct_first_file: null`; no historical score carries over.
Immutable report SHA-256:
`7261223ba479ff1c319ded187fcd6e7cf37d351ec9d62437e26b63017d1bccc2`;
scored artifact SHA-256:
`361d5fbaf917363bfedd9c3ba2ea368d1714cf28f1bc34544a9ed324761eb4db`;
preflight SHA-256:
`97c603b3078831ce0fcf19f0ebb9e15160ee2363b488946c992e881d5509ce6b`.
This failed speech lifecycle requires investigation before any live-window
timing claim; no policy, labels or sources were changed after this result.