searchi/tools/speech-eval/README.md

# Actual local speech diagnostic

Research harness only; transcription is not implemented in searchi yet.
See [the integration proposal](../../docs/SPEECH-ALIGNMENT.md). No actual results
are checked in until genuine runs complete. Google remains the semantic model.

`run.py` requires separately provisioned, pinned whisper.cpp v1.9.5 CLI, multilingual
base/small model and existing verified FFmpeg/FFprobe. It never builds/downloads.
Record the CLI build SHA beforehand and pass that hash; a matching hash alone does
not establish upstream provenance or dependency closure. Verify the build recipe
and dynamic libraries separately. CPU slot coordination is required before runs.

```powershell
$env:SEARCHI_FORCE_CPU = '1'
python tools/speech-eval/run.py --whisper PATH_TO_PINNED_CLI --whisper-sha256 RECORDED_BUILD_SHA --ffmpeg PATH_TO_VERIFIED_FFMPEG --ffprobe PATH_TO_VERIFIED_FFPROBE --model PATH_TO_VERIFIED_BASE --candidate base --scratch-parent .searchi/speech-scratch --output .searchi/qa/speech-base.json
python -m unittest discover -s tools/speech-eval -p 'test_*.py'
```

The CLI receives pinned-source-confirmed `--no-gpu`, one thread and one processor.
Source and model SHA/size are checked; NASA, both real instrumental music assets
and both non-speech effects come from unchanged media-v1. Each source is limited
to ten minutes for this proof, decoded into 30-second 16kHz mono signed-16 WAV
windows with bounded owned scratch. CLI processes restart per window: cost includes
model load, not retained-session throughput. This deliberately simple experiment
does not implement overlap/dedup or production budgets. Segment JSON offsets are
already milliseconds and get the window offset added exactly once.

Each completed window writes an incomplete checkpoint; the final report marks
complete evaluation, with `quality_failed` and explicit `validation_failures` for
invalid ASR timestamps. Bounded raw ASR JSON remains in each checkpoint before
validity is decided; invalid times are never clamped into evidence, and later
windows continue. The final report records the executed harness SHA. Process
errors retain bounded owned logs and exit/time diagnostics outside temporary scratch, which cleans
decoded WAV/JSON on exit. Cancellation terminates only owned subprocess trees.
Limits are supervisory checks, not OS resource sandbox guarantees. Recognized text
is private local diagnostic data; never commit raw logs/transcripts by default.
Music/non-speech output requires actual audio review: nonempty recognized text is
not automatically proof of hallucination and empty output is not universal safety.

`score.py` consumes **externally ranked genuine Google transcript results**, using
the existing eight frozen NASA intervals unchanged. Input has `fixture_sha256`,
`asr_report_sha256`, `google_worker_sha256`, and `rankings` mapping query ids to
ordered `{asset,start_ms,end_ms}` records. It reports top1 overlap and strict start
point hits; it never chooses a segment by looking at the answer intervals.
Hash fields must be checked against retained artifacts by the evaluator; they are
provenance declarations, not a replacement for inspecting those artifacts.

Unit tests use explicitly synthetic structural data to verify bounds/units and
scoring behavior. They provide no speech-recognition or search quality evidence.
The first proof must manually inspect NASA recognition against independently
published captions, then run real Google ranking. Eight paired queries from one
English recording do not validate German speech or general transcription quality.

Pinned CLI flag/output reference:
[upstream cli.cpp](https://github.com/ggml-org/whisper.cpp/blob/d1be6fde11ac6e0407606b4e42fe72d34add8037/examples/cli/cli.cpp).