130 lines
8.8 KiB
Markdown
130 lines
8.8 KiB
Markdown
# Actual local speech diagnostic
|
||
|
||
Research harness only; transcription is not implemented in searchi yet.
|
||
See [the integration proposal](../../docs/SPEECH-ALIGNMENT.md). No actual results
|
||
are checked in until genuine runs complete. Google remains the semantic model.
|
||
|
||
`run.py` requires separately provisioned, pinned whisper.cpp v1.9.5 CLI, multilingual
|
||
base/small model and existing verified FFmpeg/FFprobe. It never builds/downloads.
|
||
Record the CLI build SHA beforehand and pass that hash; a matching hash alone does
|
||
not establish upstream provenance or dependency closure. Verify the build recipe
|
||
and dynamic libraries separately. CPU slot coordination is required before runs.
|
||
|
||
```powershell
|
||
$env:SEARCHI_FORCE_CPU = '1'
|
||
python tools/speech-eval/run.py --whisper PATH_TO_PINNED_CLI --whisper-sha256 RECORDED_BUILD_SHA --ffmpeg PATH_TO_VERIFIED_FFMPEG --ffprobe PATH_TO_VERIFIED_FFPROBE --model PATH_TO_VERIFIED_BASE --candidate base --scratch-parent .searchi/speech-scratch --output .searchi/qa/speech-base.json
|
||
python -m unittest discover -s tools/speech-eval -p 'test_*.py'
|
||
```
|
||
|
||
The CLI receives pinned-source-confirmed `--no-gpu`, one thread and one processor.
|
||
Source and model SHA/size are checked; NASA, both real instrumental music assets
|
||
and both non-speech effects come from unchanged media-v1. Each source is limited
|
||
to ten minutes for this proof, decoded into 30-second 16kHz mono signed-16 WAV
|
||
windows with bounded owned scratch. CLI processes restart per window: cost includes
|
||
model load, not retained-session throughput. This deliberately simple experiment
|
||
does not implement overlap/dedup or production budgets. Segment JSON offsets are
|
||
already milliseconds and get the window offset added exactly once.
|
||
|
||
Windows follow the probed primary audio stream's start/duration/index, with
|
||
explicit absolute `-seek_timestamp 1` seeking and stream mapping. Missing audio
|
||
stream duration is rejected with an actionable error. Container duration cannot
|
||
silently extend a shorter audio stream. Resampling occurs before sample trimming;
|
||
decoded frames must match the planned duration within at most one millisecond.
|
||
Every actual frame count and full probe JSON is retained in the report.
|
||
|
||
Each completed window writes an incomplete checkpoint; the final report marks
|
||
complete evaluation, with `quality_failed` and explicit `validation_failures` for
|
||
invalid ASR timestamps. Bounded raw ASR JSON remains in each checkpoint before
|
||
validity is decided; invalid times are never clamped into evidence, and later
|
||
windows continue. The final report records the executed harness SHA. Process
|
||
errors retain bounded owned logs and exit/time diagnostics outside temporary scratch, which cleans
|
||
decoded WAV/JSON on exit. Cancellation terminates only owned subprocess trees.
|
||
Limits are supervisory checks, not OS resource sandbox guarantees. Recognized text
|
||
is private local diagnostic data; never commit raw logs/transcripts by default.
|
||
Music/non-speech output requires actual audio review: nonempty recognized text is
|
||
not automatically proof of hallucination and empty output is not universal safety.
|
||
|
||
`score.py` consumes **externally ranked genuine Google transcript results**, using
|
||
the existing eight frozen NASA intervals unchanged. Input has `fixture_sha256`,
|
||
`asr_report_sha256`, `google_worker_sha256`, and `rankings` mapping query ids to
|
||
ordered `{asset,start_ms,end_ms}` records. It reports top1 overlap and strict start
|
||
point hits; it never chooses a segment by looking at the answer intervals.
|
||
The ranking report must have `complete=true`, `quality_score_eligible=true` and
|
||
no excluded windows; a reduced competitor pool cannot receive an accepted score.
|
||
Hash fields must be checked against retained artifacts by the evaluator; they are
|
||
provenance declarations, not a replacement for inspecting those artifacts.
|
||
|
||
Unit tests use explicitly synthetic structural data to verify bounds/units and
|
||
scoring behavior. They provide no speech-recognition or search quality evidence.
|
||
The first proof must manually inspect NASA recognition against independently
|
||
published captions, then run real Google ranking. Eight paired queries from one
|
||
English recording do not validate German speech or general transcription quality.
|
||
|
||
Pinned CLI flag/output reference:
|
||
[upstream cli.cpp](https://github.com/ggml-org/whisper.cpp/blob/d1be6fde11ac6e0407606b4e42fe72d34add8037/examples/cli/cli.cpp).
|
||
|
||
## Frozen ordinary token-time comparison
|
||
|
||
`token-policy.json` declares an internal base-only comparison before inference.
|
||
Add `--token-sentences` to the existing command. The worker uses full JSON,
|
||
ordinary token timestamps and unchanged recognition flags; DTW is disabled.
|
||
The harness retains original raw JSON and segments, then adds exact recognized
|
||
sentence substrings and their token intervals in `aligned_segments`. No label
|
||
selects a sentence, source, word or start. All five sources and complete original
|
||
window coverage remain mandatory; empty speech is valid.
|
||
|
||
Structural tokens cannot become passage text. Ordinary text-token times must be
|
||
present, ordered and inside the source segment/window; reconstructed token text
|
||
must match original recognized bytes. Invalid alignment is retained as an error
|
||
and invalidates an accepted score. No clipping, invented time or omitted failing
|
||
competitor is permitted. Rankers must independently reconstruct these passages
|
||
from raw JSON and verify policy/contract provenance before publishing quality.
|
||
|
||
```powershell
|
||
$env:SEARCHI_FORCE_CPU = '1'
|
||
python tools/speech-eval/run.py --whisper .searchi/speech/build-cpu/bin/Release/whisper-cli.exe --whisper-sha256 12c681b4bcd9c7679b647b9a041e121a2f9e72cb2a9c6f189f529ec8b79c36f8 --ffmpeg .searchi/media-runtime/bin/ffmpeg.exe --ffprobe .searchi/media-runtime/bin/ffprobe.exe --model .searchi/speech/models/ggml-base.bin --candidate base --token-sentences --scratch-parent .searchi/speech/token-scratch --output .searchi/speech/base-token-sentences-v1.json
|
||
```
|
||
|
||
Run only with the coordinator's available CPU inference slot. The previous
|
||
base-only run took about 251 seconds including decode/process startup. Token
|
||
alignment has not yet been timed; each ASR process retains the 600-second limit.
|
||
No production behavior or quality claim follows from structural tests.
|
||
|
||
## Separate DTW point-span comparison
|
||
|
||
`dtw-policy.json` freezes a separate base experiment. Replace `--token-sentences`
|
||
with `--dtw-sentences` and use a new output such as
|
||
`.searchi/speech/base-dtw-sentences-v1.json`. This adds `--dtw base
|
||
--no-flash-attn`: the pinned runtime disables DTW when flash attention is on.
|
||
Native `t_dtw` is a centisecond point, approximately the moment a token is output,
|
||
not a start/end interval. Multiply by ten; first and last ordinary-token points
|
||
define an approximate sentence point span. One-token sentences can have zero span.
|
||
Only source-window-bounded globally monotonic integer points are accepted.
|
||
Original ordinary interval errors are retained separately and never repaired.
|
||
Missing/invalid DTW points invalidate accepted scoring for the new comparison.
|
||
Frozen ordinary-v1 output remains failed and unchanged.
|
||
|
||
Pinned native default DTW graph workspace is 128 MiB; this is additional workspace,
|
||
not a total memory ceiling. Attention storage and disabled flash attention can
|
||
increase memory/time. Input remains 30 seconds, one CPU thread, 4 MiB output,
|
||
4096 tokens/segment and 600 seconds/process; observed peak memory/runtime must
|
||
be measured before production use. No inference has established this policy.
|
||
Future harness reports capture the executing-source SHA at startup and preload
|
||
alignment helpers before processing. The first ordinary-v1 run read its harness
|
||
SHA at completion while the on-disk harness had changed; that field cannot prove
|
||
the exact executing harness bytes. Its original raw token failures remain valid
|
||
diagnostic observations, and its excluded window already prohibits an accepted
|
||
quality score. Preserve that historical limitation instead of rewriting output.
|
||
The first DTW run stopped on strict UTF-8 parsing in the second piano window
|
||
(offset 30000 ms): native JSON bytes at 1137–1138 were invalid. Its temporary
|
||
raw output was not retained by that harness, so exact malformed bytes/native
|
||
cause cannot be established. The partial checkpoint retains all three NASA
|
||
windows (21 accepted DTW sentence spans, including one independently recorded
|
||
ordinary-interval failure) and the first piano window. Missing remaining windows
|
||
and sources prohibit any accepted retrieval score. It was not rerun or decoded
|
||
lossily. Subsequent harnesses retain bounded exact failing output bytes under
|
||
the owned scratch parent with SHA/size/parse position, and atomically record a
|
||
failed incomplete checkpoint before terminating; they do not skip competitors.
|
||
References: pinned [token data/API](https://github.com/ggml-org/whisper.cpp/blob/d1be6fde11ac6e0407606b4e42fe72d34add8037/include/whisper.h)
|
||
and [DTW implementation](https://github.com/ggml-org/whisper.cpp/blob/d1be6fde11ac6e0407606b4e42fe72d34add8037/src/whisper.cpp).
|