Publish first usable Windows release instructions and validation

This commit is contained in:
karl committed 2026-10-13 11:20:23 +02:00
1 parent 5eeba29c0d
commit eabb1fbb54
9 files changed
+490 -19

No files matched your search

+35
View File
@@ -92,6 +92,41 @@ Measured hybrid p95 was **146.83 ms** on this run. [Recorded output](../fixtures
## Scoring and release gates
### Local recognized-speech supplement: actual Google diagnostic
A subsequent internal experiment transcribed the unchanged NASA, piano, violin,
bark and roaring-effect sources locally, then ranked actual recognized passages
with Google EmbeddingGemma 2 on CPU, one thread. The base candidate supplied all
30 valid segments from all five sources; music descriptions and hallucinated
words remained competitors. Exact cosine ranking over source-preserving Google
tokenizer windows, with a maximum per original segment, found the right file for
**8/8** frozen NASA queries, but achieved **4/8 strict timestamp start hits** and
**6/8 segment-overlap hits**. The Moon statement was merged into a segment starting
at zero; the orange-soil exchange was missing. Funding and Mars passage starts
passed their independent intervals.
This is a direct recognized-text retrieval diagnostic, **not the production
ANN/hybrid raw-audio pipeline**. The older production 0/8 point-hit result and
these 4/8 hits are not equivalent head-to-head measurements and cannot establish
a deployed improvement. The tiny paired English/German query set addresses four
topics in one English recording; German speech and general ASR quality remain
unverified. No model default, calibrated confidence or improved rating follows.
The small candidate's Google diagnostic completed but excluded two invalid ASR
timestamp windows (piano tail and bark), marking
`quality_score_eligible=false`. Its reduced competitor pool receives **no
accepted numeric quality score**. The base score report SHA is
`8f92c8eeed41db7e78be0e188500e441d53b4fa49fa8c0353a77df74643ce895`;
actual Google ranking reports are SHA
`2fbc860f992c3ec561f54137e30744b00265455792e360010ab1fc78ab25173a`
(base) and
`aca99d36b3bdf013b1a1074c33f7232de1c1b79af62d00236d53267ce510c9c1`
(small). Raw recognized text remains ignored. See
[speech alignment evidence](SPEECH-ALIGNMENT.md) for worker/model provenance,
corrected decoder validation, measured transcription cost and explicit failures.
Production integration, speech gating/alignment and an equivalent pipeline rerun
remain open; the experimental media self-rating stays **4/10**.
Report recall@1/@5, MRR, language/type breakdowns, segment-overlap hits, precise timestamp point hits, and failure categories. No-answer return rate is distinct from positive recall. Do not invent an arbitrary cosine rejection threshold from these results; calibrate on a separate development set, then validate on held-out negatives.
Self-rating rubric is declared before scores: **1–2/10** broken core retrieval; **3–4/10** experimental retrieval with missing essential capabilities or unvalidated abstention/metadata; **5–6/10** useful limited beta after independent basic retrieval/filter/metadata checks; **7–8/10** strong pilot with high recall and timestamp quality on representative corpora; **9–10/10** requires production release evidence across all operating systems, large representative libraries, calibrated no-answer behavior, and complete lifecycle/security/resource verification. A small diagnostic alone cannot justify a production score.