6.1 KiB
Actual local speech diagnostic
Research harness only; transcription is not implemented in searchi yet. See the integration proposal. No actual results are checked in until genuine runs complete. Google remains the semantic model.
run.py requires separately provisioned, pinned whisper.cpp v1.9.5 CLI, multilingual
base/small model and existing verified FFmpeg/FFprobe. It never builds/downloads.
Record the CLI build SHA beforehand and pass that hash; a matching hash alone does
not establish upstream provenance or dependency closure. Verify the build recipe
and dynamic libraries separately. CPU slot coordination is required before runs.
$env:SEARCHI_FORCE_CPU = '1'
python tools/speech-eval/run.py --whisper PATH_TO_PINNED_CLI --whisper-sha256 RECORDED_BUILD_SHA --ffmpeg PATH_TO_VERIFIED_FFMPEG --ffprobe PATH_TO_VERIFIED_FFPROBE --model PATH_TO_VERIFIED_BASE --candidate base --scratch-parent .searchi/speech-scratch --output .searchi/qa/speech-base.json
python -m unittest discover -s tools/speech-eval -p 'test_*.py'
The CLI receives pinned-source-confirmed --no-gpu, one thread and one processor.
Source and model SHA/size are checked; NASA, both real instrumental music assets
and both non-speech effects come from unchanged media-v1. Each source is limited
to ten minutes for this proof, decoded into 30-second 16kHz mono signed-16 WAV
windows with bounded owned scratch. CLI processes restart per window: cost includes
model load, not retained-session throughput. This deliberately simple experiment
does not implement overlap/dedup or production budgets. Segment JSON offsets are
already milliseconds and get the window offset added exactly once.
Windows follow the probed primary audio stream's start/duration/index, with
explicit absolute -seek_timestamp 1 seeking and stream mapping. Missing audio
stream duration is rejected with an actionable error. Container duration cannot
silently extend a shorter audio stream. Resampling occurs before sample trimming;
decoded frames must match the planned duration within at most one millisecond.
Every actual frame count and full probe JSON is retained in the report.
Each completed window writes an incomplete checkpoint; the final report marks
complete evaluation, with quality_failed and explicit validation_failures for
invalid ASR timestamps. Bounded raw ASR JSON remains in each checkpoint before
validity is decided; invalid times are never clamped into evidence, and later
windows continue. The final report records the executed harness SHA. Process
errors retain bounded owned logs and exit/time diagnostics outside temporary scratch, which cleans
decoded WAV/JSON on exit. Cancellation terminates only owned subprocess trees.
Limits are supervisory checks, not OS resource sandbox guarantees. Recognized text
is private local diagnostic data; never commit raw logs/transcripts by default.
Music/non-speech output requires actual audio review: nonempty recognized text is
not automatically proof of hallucination and empty output is not universal safety.
score.py consumes externally ranked genuine Google transcript results, using
the existing eight frozen NASA intervals unchanged. Input has fixture_sha256,
asr_report_sha256, google_worker_sha256, and rankings mapping query ids to
ordered {asset,start_ms,end_ms} records. It reports top1 overlap and strict start
point hits; it never chooses a segment by looking at the answer intervals.
The ranking report must have complete=true, quality_score_eligible=true and
no excluded windows; a reduced competitor pool cannot receive an accepted score.
Hash fields must be checked against retained artifacts by the evaluator; they are
provenance declarations, not a replacement for inspecting those artifacts.
Unit tests use explicitly synthetic structural data to verify bounds/units and scoring behavior. They provide no speech-recognition or search quality evidence. The first proof must manually inspect NASA recognition against independently published captions, then run real Google ranking. Eight paired queries from one English recording do not validate German speech or general transcription quality.
Pinned CLI flag/output reference: upstream cli.cpp.
Frozen ordinary token-time comparison
token-policy.json declares an internal base-only comparison before inference.
Add --token-sentences to the existing command. The worker uses full JSON,
ordinary token timestamps and unchanged recognition flags; DTW is disabled.
The harness retains original raw JSON and segments, then adds exact recognized
sentence substrings and their token intervals in aligned_segments. No label
selects a sentence, source, word or start. All five sources and complete original
window coverage remain mandatory; empty speech is valid.
Structural tokens cannot become passage text. Ordinary text-token times must be present, ordered and inside the source segment/window; reconstructed token text must match original recognized bytes. Invalid alignment is retained as an error and invalidates an accepted score. No clipping, invented time or omitted failing competitor is permitted. Rankers must independently reconstruct these passages from raw JSON and verify policy/contract provenance before publishing quality.
$env:SEARCHI_FORCE_CPU = '1'
python tools/speech-eval/run.py --whisper .searchi/speech/build-cpu/bin/Release/whisper-cli.exe --whisper-sha256 12c681b4bcd9c7679b647b9a041e121a2f9e72cb2a9c6f189f529ec8b79c36f8 --ffmpeg .searchi/media-runtime/bin/ffmpeg.exe --ffprobe .searchi/media-runtime/bin/ffprobe.exe --model .searchi/speech/models/ggml-base.bin --candidate base --token-sentences --scratch-parent .searchi/speech/token-scratch --output .searchi/speech/base-token-sentences-v1.json
Run only with the coordinator's available CPU inference slot. The previous base-only run took about 251 seconds including decode/process startup. Token alignment has not yet been timed; each ASR process retains the 600-second limit. No production behavior or quality claim follows from structural tests.