Publish first usable Windows release instructions and validation
This commit is contained in:
1 parent
5eeba29c0d
commit
eabb1fbb54
9 files changed
+490
-19
No files matched your search
@@ -92,6 +92,41 @@ Measured hybrid p95 was **146.83 ms** on this run. [Recorded output](../fixtures
|
||||
|
||||
## Scoring and release gates
|
||||
|
||||
### Local recognized-speech supplement: actual Google diagnostic
|
||||
|
||||
A subsequent internal experiment transcribed the unchanged NASA, piano, violin,
|
||||
bark and roaring-effect sources locally, then ranked actual recognized passages
|
||||
with Google EmbeddingGemma 2 on CPU, one thread. The base candidate supplied all
|
||||
30 valid segments from all five sources; music descriptions and hallucinated
|
||||
words remained competitors. Exact cosine ranking over source-preserving Google
|
||||
tokenizer windows, with a maximum per original segment, found the right file for
|
||||
**8/8** frozen NASA queries, but achieved **4/8 strict timestamp start hits** and
|
||||
**6/8 segment-overlap hits**. The Moon statement was merged into a segment starting
|
||||
at zero; the orange-soil exchange was missing. Funding and Mars passage starts
|
||||
passed their independent intervals.
|
||||
|
||||
This is a direct recognized-text retrieval diagnostic, **not the production
|
||||
ANN/hybrid raw-audio pipeline**. The older production 0/8 point-hit result and
|
||||
these 4/8 hits are not equivalent head-to-head measurements and cannot establish
|
||||
a deployed improvement. The tiny paired English/German query set addresses four
|
||||
topics in one English recording; German speech and general ASR quality remain
|
||||
unverified. No model default, calibrated confidence or improved rating follows.
|
||||
|
||||
The small candidate's Google diagnostic completed but excluded two invalid ASR
|
||||
timestamp windows (piano tail and bark), marking
|
||||
`quality_score_eligible=false`. Its reduced competitor pool receives **no
|
||||
accepted numeric quality score**. The base score report SHA is
|
||||
`8f92c8eeed41db7e78be0e188500e441d53b4fa49fa8c0353a77df74643ce895`;
|
||||
actual Google ranking reports are SHA
|
||||
`2fbc860f992c3ec561f54137e30744b00265455792e360010ab1fc78ab25173a`
|
||||
(base) and
|
||||
`aca99d36b3bdf013b1a1074c33f7232de1c1b79af62d00236d53267ce510c9c1`
|
||||
(small). Raw recognized text remains ignored. See
|
||||
[speech alignment evidence](SPEECH-ALIGNMENT.md) for worker/model provenance,
|
||||
corrected decoder validation, measured transcription cost and explicit failures.
|
||||
Production integration, speech gating/alignment and an equivalent pipeline rerun
|
||||
remain open; the experimental media self-rating stays **4/10**.
|
||||
|
||||
Report recall@1/@5, MRR, language/type breakdowns, segment-overlap hits, precise timestamp point hits, and failure categories. No-answer return rate is distinct from positive recall. Do not invent an arbitrary cosine rejection threshold from these results; calibrate on a separate development set, then validate on held-out negatives.
|
||||
|
||||
Self-rating rubric is declared before scores: **1–2/10** broken core retrieval; **3–4/10** experimental retrieval with missing essential capabilities or unvalidated abstention/metadata; **5–6/10** useful limited beta after independent basic retrieval/filter/metadata checks; **7–8/10** strong pilot with high recall and timestamp quality on representative corpora; **9–10/10** requires production release evidence across all operating systems, large representative libraries, calibrated no-answer behavior, and complete lifecycle/security/resource verification. A small diagnostic alone cannot justify a production score.
|
||||
|
||||
Reference in new issue
Block a user