Research harness only; transcription is not implemented in searchi yet. See the integration proposal. No actual results are checked in until genuine runs complete. Google remains the semantic model.

run.py requires separately provisioned, pinned whisper.cpp v1.9.5 CLI, multilingual base/small model and existing verified FFmpeg/FFprobe. It never builds/downloads. Record the CLI build SHA beforehand and pass that hash; a matching hash alone does not establish upstream provenance or dependency closure. Verify the build recipe and dynamic libraries separately. CPU slot coordination is required before runs.

$env:SEARCHI_FORCE_CPU = '1'
python tools/speech-eval/run.py --whisper PATH_TO_PINNED_CLI --whisper-sha256 RECORDED_BUILD_SHA --ffmpeg PATH_TO_VERIFIED_FFMPEG --ffprobe PATH_TO_VERIFIED_FFPROBE --model PATH_TO_VERIFIED_BASE --candidate base --scratch-parent .searchi/speech-scratch --output .searchi/qa/speech-base.json
python -m unittest discover -s tools/speech-eval -p 'test_*.py'

The CLI receives pinned-source-confirmed --no-gpu, one thread and one processor. Source and model SHA/size are checked; NASA, both real instrumental music assets and both non-speech effects come from unchanged media-v1. Each source is limited to ten minutes for this proof, decoded into 30-second 16kHz mono signed-16 WAV windows with bounded owned scratch. CLI processes restart per window: cost includes model load, not retained-session throughput. This deliberately simple experiment does not implement overlap/dedup or production budgets. Segment JSON offsets are already milliseconds and get the window offset added exactly once.

Windows follow the probed primary audio stream's start/duration/index, with explicit absolute -seek_timestamp 1 seeking and stream mapping. Missing audio stream duration is rejected with an actionable error. Container duration cannot silently extend a shorter audio stream. Resampling occurs before sample trimming; decoded frames must match the planned duration within at most one millisecond. Every actual frame count and full probe JSON is retained in the report.

Each completed window writes an incomplete checkpoint; the final report marks complete evaluation, with quality_failed and explicit validation_failures for invalid ASR timestamps. Bounded raw ASR JSON remains in each checkpoint before validity is decided; invalid times are never clamped into evidence, and later windows continue. The final report records the executed harness SHA. Process errors retain bounded owned logs and exit/time diagnostics outside temporary scratch, which cleans decoded WAV/JSON on exit. Cancellation terminates only owned subprocess trees. Limits are supervisory checks, not OS resource sandbox guarantees. Recognized text is private local diagnostic data; never commit raw logs/transcripts by default. Music/non-speech output requires actual audio review: nonempty recognized text is not automatically proof of hallucination and empty output is not universal safety.

score.py consumes externally ranked genuine Google transcript results, using the existing eight frozen NASA intervals unchanged. Input has fixture_sha256, asr_report_sha256, google_worker_sha256, and rankings mapping query ids to ordered {asset,start_ms,end_ms} records. It reports top1 overlap and strict start point hits; it never chooses a segment by looking at the answer intervals. The ranking report must have complete=true, quality_score_eligible=true and no excluded windows; a reduced competitor pool cannot receive an accepted score. Hash fields must be checked against retained artifacts by the evaluator; they are provenance declarations, not a replacement for inspecting those artifacts.

Unit tests use explicitly synthetic structural data to verify bounds/units and scoring behavior. They provide no speech-recognition or search quality evidence. The first proof must manually inspect NASA recognition against independently published captions, then run real Google ranking. Eight paired queries from one English recording do not validate German speech or general transcription quality.

Pinned CLI flag/output reference: upstream cli.cpp.

Frozen ordinary token-time comparison

token-policy.json declares an internal base-only comparison before inference. Add --token-sentences to the existing command. The worker uses full JSON, ordinary token timestamps and unchanged recognition flags; DTW is disabled. The harness retains original raw JSON and segments, then adds exact recognized sentence substrings and their token intervals in aligned_segments. No label selects a sentence, source, word or start. All five sources and complete original window coverage remain mandatory; empty speech is valid.

Structural tokens cannot become passage text. Ordinary text-token times must be present, ordered and inside the source segment/window; reconstructed token text must match original recognized bytes. Invalid alignment is retained as an error and invalidates an accepted score. No clipping, invented time or omitted failing competitor is permitted. Rankers must independently reconstruct these passages from raw JSON and verify policy/contract provenance before publishing quality.

$env:SEARCHI_FORCE_CPU = '1'
python tools/speech-eval/run.py --whisper .searchi/speech/build-cpu/bin/Release/whisper-cli.exe --whisper-sha256 12c681b4bcd9c7679b647b9a041e121a2f9e72cb2a9c6f189f529ec8b79c36f8 --ffmpeg .searchi/media-runtime/bin/ffmpeg.exe --ffprobe .searchi/media-runtime/bin/ffprobe.exe --model .searchi/speech/models/ggml-base.bin --candidate base --token-sentences --scratch-parent .searchi/speech/token-scratch --output .searchi/speech/base-token-sentences-v1.json

Run only with the coordinator's available CPU inference slot. The previous base-only run took about 251 seconds including decode/process startup. Token alignment has not yet been timed; each ASR process retains the 600-second limit. No production behavior or quality claim follows from structural tests.

Separate DTW point-span comparison

dtw-policy.json freezes a separate base experiment. Replace --token-sentences with --dtw-sentences and use a new output such as .searchi/speech/base-dtw-sentences-v1.json. This adds --dtw base --no-flash-attn: the pinned runtime disables DTW when flash attention is on. Native t_dtw is a centisecond point, approximately the moment a token is output, not a start/end interval. Multiply by ten; first and last ordinary-token points define an approximate sentence point span. One-token sentences can have zero span. Only source-window-bounded globally monotonic integer points are accepted. Original ordinary interval errors are retained separately and never repaired. Missing/invalid DTW points invalidate accepted scoring for the new comparison. Frozen ordinary-v1 output remains failed and unchanged.

Pinned native default DTW graph workspace is 128 MiB; this is additional workspace, not a total memory ceiling. Attention storage and disabled flash attention can increase memory/time. Input remains 30 seconds, one CPU thread, 4 MiB output, 4096 tokens/segment and 600 seconds/process; observed peak memory/runtime must be measured before production use. No inference has established this policy. Future harness reports capture the executing-source SHA at startup and preload alignment helpers before processing. The first ordinary-v1 run read its harness SHA at completion while the on-disk harness had changed; that field cannot prove the exact executing harness bytes. Its original raw token failures remain valid diagnostic observations, and its excluded window already prohibits an accepted quality score. Preserve that historical limitation instead of rewriting output. References: pinned token data/API and DTW implementation.