96 lines
6.1 KiB
Markdown
96 lines
6.1 KiB
Markdown
# Genuine Google transcript ranking diagnostic
|
|
|
|
This internal experiment ranks **actual recognized ASR text** with the shared
|
|
Google EmbeddingGemma 2 adapter. It is separate from production ANN/hybrid search
|
|
and does not add speech recognition to the application.
|
|
|
|
All valid segments from the NASA video and four music/sound competitors enter
|
|
the same candidate pool. Original source filenames are the only document titles;
|
|
recognized text is unchanged. Frozen relevance intervals never select, rewrite,
|
|
filter or rank candidates. General Google search queries use the normal 512-token
|
|
query role; document encoding uses 2048 tokens. The document process is dropped
|
|
before opening the query process. Both use forced CPU and one thread.
|
|
|
|
Long segments use the actual Google tokenizer's source-preserving windows.
|
|
Every vector's original segment and UTF-8 byte range are recorded. Exact cosine
|
|
scores are computed from genuine Google vectors and aggregated by maximum over
|
|
token windows belonging to the same original ASR segment. All vector and segment
|
|
ranks are retained; at most the first 100 segment hits per query are exported for
|
|
the existing scorer. Ties use stable source order. No confidence threshold or
|
|
oracle matching is applied.
|
|
|
|
The additional frozen `gated-vad-v1` diagnostic policy accepts only the exact
|
|
VAD report SHA-256 `093be2005b57fe16a8f85161cc9da8966e70e2050cf19fa4122531e304dd70df`.
|
|
The ASR report embeds its original UTF-8 bytes and parsed document; both must
|
|
agree. Source probes, decoder hashes and the pinned base Whisper worker must
|
|
match. The ranker independently rebuilds every contiguous window from the frozen
|
|
bounded source-sample intersections, retaining interval identity and explicit
|
|
sub-millisecond tail remainder. Omitted intervals, changed windows or claimed
|
|
no-speech status without frozen empty intervals are rejected before inference.
|
|
All five sources remain explicit competitors; no relevance label selects which
|
|
source receives ASR. Existing ungated reports still require full audio coverage.
|
|
|
|
```powershell
|
|
cargo test --manifest-path tools/speech-rank/Cargo.toml --locked
|
|
cargo build --release --manifest-path tools/speech-rank/Cargo.toml --locked
|
|
$env:SEARCHI_FORCE_CPU = '1'
|
|
tools/speech-rank/target/release/searchi-speech-rank.exe .searchi/speech/base-results.json fixtures/media/fixture.json C:/path/to/verified-models C:/path/to/searchi-gemma-worker.exe .searchi/speech/base-google-ranks.json
|
|
python tools/speech-eval/score.py --rankings .searchi/speech/base-google-ranks.json --output .searchi/speech/base-google-score.json
|
|
```
|
|
|
|
Run only after the shared inference CPU slot is released. The output directory
|
|
must exist, and the output filename must be new. `create_new` reserves evidence
|
|
before costly work; atomic JSON checkpoints retain incomplete phases. Inputs
|
|
are bounded, source/model hashes verified, and recognized segment timestamps
|
|
reconstructed from actual Whisper JSON to reject stale normalized text. Reports
|
|
from truncated decoding or an older policy are rejected. The artifact records
|
|
the actual ASR bytes, worker/runner/model hashes and process/timing information.
|
|
|
|
Invalid ASR windows are explicitly marked ineligible with reasons, never quietly
|
|
discarded. A report with any excluded window has `quality_score_eligible: false`:
|
|
do not publish a quality score from a reduced competitor pool. Valid hallucinated
|
|
music or sound text remains eligible and must compete. `complete` reports prove
|
|
the diagnostic finished, not speech accuracy.
|
|
|
|
Exported `start_ms`/`end_ms` refer to the full original media timeline. They are
|
|
ASR segment approximations, potentially affected by 30-second recognition
|
|
boundaries; they are **not word alignment**. A token window retains its containing
|
|
segment's same interval. The existing eight frozen NASA queries constitute a tiny
|
|
diagnostic, not held-out calibration, general media quality or production proof.
|
|
|
|
The optional frozen `ordinary-token-sentences-v1` input retains those original
|
|
segments and supplies full Whisper JSON. The ranker independently reconstructs
|
|
exact recognized sentences from ordinary token times, excludes structural tokens
|
|
from passage text, and verifies the declared policy and Python contract hashes.
|
|
Changed text, missing/out-of-range/unordered timestamps, forged normalized
|
|
passages, omitted windows or partial provenance are rejected. All five sources
|
|
remain competitors. Real alignment failures are retained as excluded windows;
|
|
such a report cannot receive an accepted quality score. The old ungated/gated
|
|
policies are unchanged. Token times remain approximate recognized-speech
|
|
coordinates; this experiment does not make them verified quotations or add
|
|
production transcription.
|
|
|
|
After the complete base token-sentence ASR run and coordinator CPU-slot grant,
|
|
use the existing executable command with input
|
|
`.searchi/speech/base-token-sentences-v1.json` and a new output filename, then the
|
|
same frozen eight-query scorer. Do not overwrite old results or select passages
|
|
with relevance labels.
|
|
|
|
The separate `dtw-token-sentences-v1` experiment independently validates native
|
|
DTW centisecond points and uses the first/last token points as approximate
|
|
sentence spans. These are not token durations. Ordinary interval failures remain
|
|
in the embedded ASR evidence and are independently checked; they do not authorize
|
|
fabricating or clamping DTW points. Missing/invalid DTW points exclude a window
|
|
and prevent accepted scoring. Use a separate DTW report/output and retain the
|
|
failed ordinary-v1 run. The frozen five sources/eight queries remain unchanged.
|
|
|
|
The separately frozen `gated-dtw-v1` combination embeds exact manifest bytes
|
|
with SHA `d3bf453f21dac03fd029344af2fe700480cf686d2ba5b5d7fcc79134dda1c563`.
|
|
The ranker verifies component hashes and the original frozen VAD bytes, rebuilds
|
|
all twelve speech windows from source intersections, requires explicit no-speech
|
|
status for empty competitors, then independently reconstructs DTW points. All
|
|
five sources remain mandatory; dropped windows, fabricated no-speech claims,
|
|
tampered policy bytes and invalid points are rejected. This combination leaves
|
|
the old gated base and ungated DTW policies unchanged. Structural token tests
|
|
provide no new quality result; actual model inference remains a separate gate.
|