Validate audio stream timelines and add genuine Google transcript diagnostics

tools/speech-rank/README.md

@@ -0,0 +1,48 @@
+# Genuine Google transcript ranking diagnostic
+
+This internal experiment ranks **actual recognized ASR text** with the shared
+Google EmbeddingGemma 2 adapter. It is separate from production ANN/hybrid search
+and does not add speech recognition to the application.
+
+All valid segments from the NASA video and four music/sound competitors enter
+the same candidate pool. Original source filenames are the only document titles;
+recognized text is unchanged. Frozen relevance intervals never select, rewrite,
+filter or rank candidates. General Google search queries use the normal 512-token
+query role; document encoding uses 2048 tokens. The document process is dropped
+before opening the query process. Both use forced CPU and one thread.
+
+Long segments use the actual Google tokenizer's source-preserving windows.
+Every vector's original segment and UTF-8 byte range are recorded. Exact cosine
+scores are computed from genuine Google vectors and aggregated by maximum over
+token windows belonging to the same original ASR segment. All vector and segment
+ranks are retained; at most the first 100 segment hits per query are exported for
+the existing scorer. Ties use stable source order. No confidence threshold or
+oracle matching is applied.
+
+```powershell
+cargo test --manifest-path tools/speech-rank/Cargo.toml --locked
+cargo build --release --manifest-path tools/speech-rank/Cargo.toml --locked
+$env:SEARCHI_FORCE_CPU = '1'
+tools/speech-rank/target/release/searchi-speech-rank.exe .searchi/speech/base-results.json fixtures/media/fixture.json C:/path/to/verified-models C:/path/to/searchi-gemma-worker.exe .searchi/speech/base-google-ranks.json
+python tools/speech-eval/score.py --rankings .searchi/speech/base-google-ranks.json --output .searchi/speech/base-google-score.json
+```
+
+Run only after the shared inference CPU slot is released. The output directory
+must exist, and the output filename must be new. `create_new` reserves evidence
+before costly work; atomic JSON checkpoints retain incomplete phases. Inputs
+are bounded, source/model hashes verified, and recognized segment timestamps
+reconstructed from actual Whisper JSON to reject stale normalized text. Reports
+from truncated decoding or an older policy are rejected. The artifact records
+the actual ASR bytes, worker/runner/model hashes and process/timing information.
+
+Invalid ASR windows are explicitly marked ineligible with reasons, never quietly
+discarded. A report with any excluded window has `quality_score_eligible: false`:
+do not publish a quality score from a reduced competitor pool. Valid hallucinated
+music or sound text remains eligible and must compete. `complete` reports prove
+the diagnostic finished, not speech accuracy.
+
+Exported `start_ms`/`end_ms` refer to the full original media timeline. They are
+ASR segment approximations, potentially affected by 30-second recognition
+boundaries; they are **not word alignment**. A token window retains its containing
+segment's same interval. The existing eight frozen NASA queries constitute a tiny
+diagnostic, not held-out calibration, general media quality or production proof.