120 lines
8.0 KiB
Markdown
120 lines
8.0 KiB
Markdown
# Actual local speech diagnostic
|
|
|
|
Research harness only; transcription is not implemented in searchi yet.
|
|
See [the integration proposal](../../docs/SPEECH-ALIGNMENT.md). No actual results
|
|
are checked in until genuine runs complete. Google remains the semantic model.
|
|
|
|
`run.py` requires separately provisioned, pinned whisper.cpp v1.9.5 CLI, multilingual
|
|
base/small model and existing verified FFmpeg/FFprobe. It never builds/downloads.
|
|
Record the CLI build SHA beforehand and pass that hash; a matching hash alone does
|
|
not establish upstream provenance or dependency closure. Verify the build recipe
|
|
and dynamic libraries separately. CPU slot coordination is required before runs.
|
|
|
|
```powershell
|
|
$env:SEARCHI_FORCE_CPU = '1'
|
|
python tools/speech-eval/run.py --whisper PATH_TO_PINNED_CLI --whisper-sha256 RECORDED_BUILD_SHA --ffmpeg PATH_TO_VERIFIED_FFMPEG --ffprobe PATH_TO_VERIFIED_FFPROBE --model PATH_TO_VERIFIED_BASE --candidate base --scratch-parent .searchi/speech-scratch --output .searchi/qa/speech-base.json
|
|
python -m unittest discover -s tools/speech-eval -p 'test_*.py'
|
|
```
|
|
|
|
The CLI receives pinned-source-confirmed `--no-gpu`, one thread and one processor.
|
|
Source and model SHA/size are checked; NASA, both real instrumental music assets
|
|
and both non-speech effects come from unchanged media-v1. Each source is limited
|
|
to ten minutes for this proof, decoded into 30-second 16kHz mono signed-16 WAV
|
|
windows with bounded owned scratch. CLI processes restart per window: cost includes
|
|
model load, not retained-session throughput. This deliberately simple experiment
|
|
does not implement overlap/dedup or production budgets. Segment JSON offsets are
|
|
already milliseconds and get the window offset added exactly once.
|
|
|
|
Windows follow the probed primary audio stream's start/duration/index, with
|
|
explicit absolute `-seek_timestamp 1` seeking and stream mapping. Missing audio
|
|
stream duration is rejected with an actionable error. Container duration cannot
|
|
silently extend a shorter audio stream. Resampling occurs before sample trimming;
|
|
decoded frames must match the planned duration within at most one millisecond.
|
|
Every actual frame count and full probe JSON is retained in the report.
|
|
|
|
Each completed window writes an incomplete checkpoint; the final report marks
|
|
complete evaluation, with `quality_failed` and explicit `validation_failures` for
|
|
invalid ASR timestamps. Bounded raw ASR JSON remains in each checkpoint before
|
|
validity is decided; invalid times are never clamped into evidence, and later
|
|
windows continue. The final report records the executed harness SHA. Process
|
|
errors retain bounded owned logs and exit/time diagnostics outside temporary scratch, which cleans
|
|
decoded WAV/JSON on exit. Cancellation terminates only owned subprocess trees.
|
|
Limits are supervisory checks, not OS resource sandbox guarantees. Recognized text
|
|
is private local diagnostic data; never commit raw logs/transcripts by default.
|
|
Music/non-speech output requires actual audio review: nonempty recognized text is
|
|
not automatically proof of hallucination and empty output is not universal safety.
|
|
|
|
`score.py` consumes **externally ranked genuine Google transcript results**, using
|
|
the existing eight frozen NASA intervals unchanged. Input has `fixture_sha256`,
|
|
`asr_report_sha256`, `google_worker_sha256`, and `rankings` mapping query ids to
|
|
ordered `{asset,start_ms,end_ms}` records. It reports top1 overlap and strict start
|
|
point hits; it never chooses a segment by looking at the answer intervals.
|
|
The ranking report must have `complete=true`, `quality_score_eligible=true` and
|
|
no excluded windows; a reduced competitor pool cannot receive an accepted score.
|
|
Hash fields must be checked against retained artifacts by the evaluator; they are
|
|
provenance declarations, not a replacement for inspecting those artifacts.
|
|
|
|
Unit tests use explicitly synthetic structural data to verify bounds/units and
|
|
scoring behavior. They provide no speech-recognition or search quality evidence.
|
|
The first proof must manually inspect NASA recognition against independently
|
|
published captions, then run real Google ranking. Eight paired queries from one
|
|
English recording do not validate German speech or general transcription quality.
|
|
|
|
Pinned CLI flag/output reference:
|
|
[upstream cli.cpp](https://github.com/ggml-org/whisper.cpp/blob/d1be6fde11ac6e0407606b4e42fe72d34add8037/examples/cli/cli.cpp).
|
|
|
|
## Frozen ordinary token-time comparison
|
|
|
|
`token-policy.json` declares an internal base-only comparison before inference.
|
|
Add `--token-sentences` to the existing command. The worker uses full JSON,
|
|
ordinary token timestamps and unchanged recognition flags; DTW is disabled.
|
|
The harness retains original raw JSON and segments, then adds exact recognized
|
|
sentence substrings and their token intervals in `aligned_segments`. No label
|
|
selects a sentence, source, word or start. All five sources and complete original
|
|
window coverage remain mandatory; empty speech is valid.
|
|
|
|
Structural tokens cannot become passage text. Ordinary text-token times must be
|
|
present, ordered and inside the source segment/window; reconstructed token text
|
|
must match original recognized bytes. Invalid alignment is retained as an error
|
|
and invalidates an accepted score. No clipping, invented time or omitted failing
|
|
competitor is permitted. Rankers must independently reconstruct these passages
|
|
from raw JSON and verify policy/contract provenance before publishing quality.
|
|
|
|
```powershell
|
|
$env:SEARCHI_FORCE_CPU = '1'
|
|
python tools/speech-eval/run.py --whisper .searchi/speech/build-cpu/bin/Release/whisper-cli.exe --whisper-sha256 12c681b4bcd9c7679b647b9a041e121a2f9e72cb2a9c6f189f529ec8b79c36f8 --ffmpeg .searchi/media-runtime/bin/ffmpeg.exe --ffprobe .searchi/media-runtime/bin/ffprobe.exe --model .searchi/speech/models/ggml-base.bin --candidate base --token-sentences --scratch-parent .searchi/speech/token-scratch --output .searchi/speech/base-token-sentences-v1.json
|
|
```
|
|
|
|
Run only with the coordinator's available CPU inference slot. The previous
|
|
base-only run took about 251 seconds including decode/process startup. Token
|
|
alignment has not yet been timed; each ASR process retains the 600-second limit.
|
|
No production behavior or quality claim follows from structural tests.
|
|
|
|
## Separate DTW point-span comparison
|
|
|
|
`dtw-policy.json` freezes a separate base experiment. Replace `--token-sentences`
|
|
with `--dtw-sentences` and use a new output such as
|
|
`.searchi/speech/base-dtw-sentences-v1.json`. This adds `--dtw base
|
|
--no-flash-attn`: the pinned runtime disables DTW when flash attention is on.
|
|
Native `t_dtw` is a centisecond point, approximately the moment a token is output,
|
|
not a start/end interval. Multiply by ten; first and last ordinary-token points
|
|
define an approximate sentence point span. One-token sentences can have zero span.
|
|
Only source-window-bounded globally monotonic integer points are accepted.
|
|
Original ordinary interval errors are retained separately and never repaired.
|
|
Missing/invalid DTW points invalidate accepted scoring for the new comparison.
|
|
Frozen ordinary-v1 output remains failed and unchanged.
|
|
|
|
Pinned native default DTW graph workspace is 128 MiB; this is additional workspace,
|
|
not a total memory ceiling. Attention storage and disabled flash attention can
|
|
increase memory/time. Input remains 30 seconds, one CPU thread, 4 MiB output,
|
|
4096 tokens/segment and 600 seconds/process; observed peak memory/runtime must
|
|
be measured before production use. No inference has established this policy.
|
|
Future harness reports capture the executing-source SHA at startup and preload
|
|
alignment helpers before processing. The first ordinary-v1 run read its harness
|
|
SHA at completion while the on-disk harness had changed; that field cannot prove
|
|
the exact executing harness bytes. Its original raw token failures remain valid
|
|
diagnostic observations, and its excluded window already prohibits an accepted
|
|
quality score. Preserve that historical limitation instead of rewriting output.
|
|
References: pinned [token data/API](https://github.com/ggml-org/whisper.cpp/blob/d1be6fde11ac6e0407606b4e42fe72d34add8037/include/whisper.h)
|
|
and [DTW implementation](https://github.com/ggml-org/whisper.cpp/blob/d1be6fde11ac6e0407606b4e42fe72d34add8037/src/whisper.cpp).
|