49 lines
3.0 KiB
Markdown
49 lines
3.0 KiB
Markdown
# Genuine Google transcript ranking diagnostic
|
|
|
|
This internal experiment ranks **actual recognized ASR text** with the shared
|
|
Google EmbeddingGemma 2 adapter. It is separate from production ANN/hybrid search
|
|
and does not add speech recognition to the application.
|
|
|
|
All valid segments from the NASA video and four music/sound competitors enter
|
|
the same candidate pool. Original source filenames are the only document titles;
|
|
recognized text is unchanged. Frozen relevance intervals never select, rewrite,
|
|
filter or rank candidates. General Google search queries use the normal 512-token
|
|
query role; document encoding uses 2048 tokens. The document process is dropped
|
|
before opening the query process. Both use forced CPU and one thread.
|
|
|
|
Long segments use the actual Google tokenizer's source-preserving windows.
|
|
Every vector's original segment and UTF-8 byte range are recorded. Exact cosine
|
|
scores are computed from genuine Google vectors and aggregated by maximum over
|
|
token windows belonging to the same original ASR segment. All vector and segment
|
|
ranks are retained; at most the first 100 segment hits per query are exported for
|
|
the existing scorer. Ties use stable source order. No confidence threshold or
|
|
oracle matching is applied.
|
|
|
|
```powershell
|
|
cargo test --manifest-path tools/speech-rank/Cargo.toml --locked
|
|
cargo build --release --manifest-path tools/speech-rank/Cargo.toml --locked
|
|
$env:SEARCHI_FORCE_CPU = '1'
|
|
tools/speech-rank/target/release/searchi-speech-rank.exe .searchi/speech/base-results.json fixtures/media/fixture.json C:/path/to/verified-models C:/path/to/searchi-gemma-worker.exe .searchi/speech/base-google-ranks.json
|
|
python tools/speech-eval/score.py --rankings .searchi/speech/base-google-ranks.json --output .searchi/speech/base-google-score.json
|
|
```
|
|
|
|
Run only after the shared inference CPU slot is released. The output directory
|
|
must exist, and the output filename must be new. `create_new` reserves evidence
|
|
before costly work; atomic JSON checkpoints retain incomplete phases. Inputs
|
|
are bounded, source/model hashes verified, and recognized segment timestamps
|
|
reconstructed from actual Whisper JSON to reject stale normalized text. Reports
|
|
from truncated decoding or an older policy are rejected. The artifact records
|
|
the actual ASR bytes, worker/runner/model hashes and process/timing information.
|
|
|
|
Invalid ASR windows are explicitly marked ineligible with reasons, never quietly
|
|
discarded. A report with any excluded window has `quality_score_eligible: false`:
|
|
do not publish a quality score from a reduced competitor pool. Valid hallucinated
|
|
music or sound text remains eligible and must compete. `complete` reports prove
|
|
the diagnostic finished, not speech accuracy.
|
|
|
|
Exported `start_ms`/`end_ms` refer to the full original media timeline. They are
|
|
ASR segment approximations, potentially affected by 30-second recognition
|
|
boundaries; they are **not word alignment**. A token window retains its containing
|
|
segment's same interval. The existing eight frozen NASA queries constitute a tiny
|
|
diagnostic, not held-out calibration, general media quality or production proof.
|