Publish first usable Windows release instructions and validation
This commit is contained in:
1 parent
5eeba29c0d
commit
eabb1fbb54
9 files changed
+490
-19
No files matched your search
+151
-7
@@ -174,9 +174,10 @@ transcripts, worker errors and inferred content never become executable instruct
|
||||
- [ ] Validate normal package dependency closure and missing-driver CPU fallback
|
||||
on each native OS; execute physical GPU tests outside this Windows VM.
|
||||
|
||||
Next implementation slice: isolated worker + manifest verification + genuine NASA
|
||||
proof. Keep the initial runtime experiment separate from schema/ranking changes
|
||||
until it establishes that recognized segments actually fix the measured problem.
|
||||
The runtime and genuine retrieval diagnostics below inform future work; they do
|
||||
not add a production speech feature. Further ASR/VAD research is deferred while
|
||||
the usable Windows desktop takes priority. Revisit gating/alignment before schema
|
||||
or ranking changes that would expose unreliable recognized speech to users.
|
||||
|
||||
## Actual internal CPU diagnostic, 2026-10-13
|
||||
|
||||
@@ -221,7 +222,150 @@ timestamp output. They do not establish speech recognition quality.
|
||||
Historical raw/checkpoint hashes and invalid classification remain under ignored
|
||||
`.searchi/speech/invalid-truncated-audio.json`; model bytes and actual raw outputs
|
||||
are excluded from Git. The owned sequential session 20331 is terminal with exit 1
|
||||
following intentional invalid-experiment termination. Corrected base/small
|
||||
inference is paused pending the coordinator's CPU slot, followed by independent
|
||||
caption review and genuine Google transcript ranking. Neither candidate is an
|
||||
accepted product default; the media quality rating remains 4/10.
|
||||
following intentional invalid-experiment termination. A subsequent corrected run
|
||||
also detected a source-timeline issue: NASA's container ends at 85.333333 seconds,
|
||||
but its primary audio ends at 85.262 seconds. Session 6691 failed closed on the
|
||||
72 ms tail discrepancy, preserving its completed checkpoint. The harness now
|
||||
uses the actual primary audio stream's duration/start/index, absolute seeking
|
||||
with `-seek_timestamp 1`, and explicit stream mapping. Missing stream duration
|
||||
is an actionable error; container duration is never silently substituted.
|
||||
|
||||
Twelve tests passed in 1.937 seconds after that correction, including actual
|
||||
FFmpeg mux/decode with a longer video, audio offset 0.125 seconds and container
|
||||
offset five seconds. The audio timeline was 5.125–36.125 seconds; its one-second
|
||||
tail decoded 15997 samples, within the unchanged one-millisecond rounding bound.
|
||||
The zero-start 44.1/48 kHz tests still decoded exactly 480000/16000 samples.
|
||||
|
||||
## Corrected base candidate evidence
|
||||
|
||||
The corrected base evaluation completed all five assets with zero timestamp or
|
||||
sample-coverage validation failures. Report SHA:
|
||||
`9e33984afac9480452f506070dee0156fcc8d7f0d6882c8f5feeeaf5cae57de2`.
|
||||
Executed frozen harness SHA:
|
||||
`16e29bb70d3e1eeaad1f5b8ec223bff1a6f359215bcf8dabe52512a451855346`.
|
||||
|
||||
| Source | Windows | Recognized segments | ASR process seconds | Decode seconds |
|
||||
| --- | ---: | ---: | ---: | ---: |
|
||||
| NASA | 3 | 19 | 23.322 | 0.542 |
|
||||
| Piano | 9 | 9 | 210.720 | 2.167 |
|
||||
| Violin | 1 | 1 | 4.471 | 0.212 |
|
||||
| Bark | 1 | 0 | 4.373 | 0.113 |
|
||||
| Roaring effect | 1 | 1 | 4.732 | 0.111 |
|
||||
|
||||
These are observed serial CPU process times including model startup, not warmed
|
||||
throughput or a controlled hardware comparison. NASA's decoded windows contained
|
||||
480000, 480000 and 404192 samples, covering the actual primary audio range.
|
||||
Manual comparison with publisher captions finds the Artemis funding and Mars
|
||||
statements recognized, but the initial Moon-choice statement is merged into a
|
||||
segment starting at zero and the orange-soil exchange is omitted. Recognition
|
||||
does not prove Google ranking or a precise timestamp success rate.
|
||||
|
||||
The instrumental piano emitted an unrelated water phrase and repetitive numbers
|
||||
or punctuation in several windows. Music descriptions also appear. These must
|
||||
remain competing retrieved candidates in the next diagnostic; filtering them
|
||||
by knowledge of labels would hide failure. The source's published instrumental
|
||||
description supports a hallucination concern, but independent audio listening
|
||||
remains open. The current report field `quality_failed=false` means only that
|
||||
structural timestamp/sample validation passed; **it does not certify speech or
|
||||
search quality**. No model default is accepted.
|
||||
|
||||
## Corrected small candidate evidence
|
||||
|
||||
Owned sequential session 89200 finished with exit zero after evaluating all five
|
||||
assets, base then small. Both used the same frozen corrected harness and verified
|
||||
model bytes. The complete small report SHA is
|
||||
`1660c95c71c9b8b1cb3b77f8f1ec422fb0954f2aacd2c16ea773901c0cfbe099`.
|
||||
|
||||
| Source | Windows | Valid recognized segments | ASR process seconds | Decode seconds |
|
||||
| --- | ---: | ---: | ---: | ---: |
|
||||
| NASA | 3 | 19 | 86.250 | 0.601 |
|
||||
| Piano | 9 | 8 | 1176.426 | 2.399 |
|
||||
| Violin | 1 | 1 | 16.045 | 0.162 |
|
||||
| Bark | 1 | 0 | 16.701 | 0.112 |
|
||||
| Roaring effect | 1 | 1 | 25.082 | 0.113 |
|
||||
|
||||
Small emitted repetitive inferred words in non-English languages on the
|
||||
instrumental piano, with substantial CPU cost. Two actual timestamp outputs
|
||||
failed validation: the 14.454-second piano tail returned a 0–30-second segment,
|
||||
and the 2.039-second bark window returned a 0–2.5-second segment. Decoded sample
|
||||
counts were correct (231254 and 32618 respectively); these are ASR timestamp
|
||||
failures, not the historical resampling bug. Raw JSON is retained; no timestamp
|
||||
was clamped. The report's `complete=true` establishes complete evaluation,
|
||||
while `quality_failed=true` records failed structural validation.
|
||||
|
||||
NASA's small recognition retained funding and Mars passages but misrecognized
|
||||
the orange-soil words and still produced coarse segment boundaries. No numeric
|
||||
timestamp retrieval improvement follows from caption inspection. The small
|
||||
ranker may retain diagnostic results, but its reduced valid-window pool must be
|
||||
`quality_score_eligible=false`; the scorer refuses an accepted numeric quality
|
||||
score for incomplete or reduced competitor pools. Thirteen harness regressions,
|
||||
including real FFmpeg coverage/timeline checks and this scoring gate, passed in
|
||||
the coordinator's verified-FFmpeg run (2.100 seconds).
|
||||
|
||||
Final process inspection found no owned ASR/runner processes; the CPU slot was
|
||||
released for genuine Google transcript ranking. The current run removed its
|
||||
temporary decoded WAV/JSON. An older scratch directory from intentionally stopped
|
||||
invalid session 20331 remains ignored: automatic approval review rejected native
|
||||
PowerShell recursive cleanup with only “blocked by policy” as its reason. No
|
||||
alternate deletion method was attempted.
|
||||
|
||||
Neither ASR model is an accepted product default. Production Google transcript integration,
|
||||
speech activity gating, tighter alignment, hallucination control, German speech,
|
||||
resource scheduling and native distribution still require evidence. Publish raw
|
||||
Google media search before completing derived transcription so slow ASR cannot
|
||||
block initial semantic readiness. Transcript publication needs its own bounded
|
||||
completed-source work and atomic eligibility update. The production media rating
|
||||
remains 4/10.
|
||||
|
||||
## Actual Google transcript retrieval diagnostic
|
||||
|
||||
The coordinator's owned session 2984 completed base and small ranking with exit
|
||||
zero. The shared real Google adapter ran on CPU, one thread: document encoding
|
||||
used the normal indexing role (2048-token context), then the document process was
|
||||
dropped before opening the query role (512 tokens). Both reported
|
||||
`google/embeddinggemma-2`, 768 dimensions and `b11461-no-logits-v1`. The diagnostic
|
||||
runner was a Debug build with SHA
|
||||
`be096c6dfe158b991520c060ca83478bdd0cd5c34d64fab4a2413d8b01de8f15`;
|
||||
the actual native Google worker SHA was
|
||||
`c01cb3ec13cfca722ee768e8350e1399a8b9a4d14a24a35adc3473a5cfc82a73`.
|
||||
|
||||
The base experiment ranked all **30 valid recognized segments from all five
|
||||
sources**, preserving music/sound descriptions and hallucinated text as
|
||||
competitors. Actual Google tokenizer windows preserved original UTF-8 source
|
||||
ranges. Segment ranking used exact cosine and maximum over its token windows;
|
||||
the frozen labels did not select or rewrite candidates. The eight unchanged
|
||||
English/German NASA queries produced:
|
||||
|
||||
| Metric | Result |
|
||||
| --- | ---: |
|
||||
| Correct file at rank one | 8/8 |
|
||||
| Strict timestamp start hit at rank one | 4/8 |
|
||||
| Segment overlap at rank one | 6/8 |
|
||||
|
||||
Both Moon-choice queries selected the merged segment starting at zero, before
|
||||
the labelled 1.551–3.603-second utterance. Both orange-soil queries selected an
|
||||
unrelated passage starting at 24.320 seconds because the recognized exchange
|
||||
was missing. Funding queries selected 43.920 seconds and Mars queries selected
|
||||
60 seconds, within the independently frozen intervals. Correct-file success
|
||||
does not establish precise spoken-passage localization.
|
||||
|
||||
The complete base Google report SHA is
|
||||
`2fbc860f992c3ec561f54137e30744b00265455792e360010ab1fc78ab25173a`;
|
||||
the accepted diagnostic scoring artifact SHA is
|
||||
`8f92c8eeed41db7e78be0e188500e441d53b4fa49fa8c0353a77df74643ce895`.
|
||||
Raw artifacts remain ignored under `.searchi/speech`. This is **direct exact-cosine
|
||||
retrieval of recognized text**, not production ANN/hybrid retrieval of raw audio.
|
||||
The historical production 0/8 audio-window result is useful motivation but is
|
||||
not an equivalent head-to-head experiment; no production improvement or score
|
||||
change is claimed. Eight correlated queries about four topics in one English
|
||||
video do not validate German speech, held-out relevance or abstention.
|
||||
|
||||
The small diagnostic also completed, with report SHA
|
||||
`aca99d36b3bdf013b1a1074c33f7232de1c1b79af62d00236d53267ce510c9c1`.
|
||||
Its two invalid ASR windows (piano tail and bark) were explicitly excluded:
|
||||
`complete=true`, `quality_score_eligible=false`, `excluded_window_count=2`.
|
||||
Diagnostic ranks are retained, but **no accepted numeric quality score** is
|
||||
published for that reduced competitor pool. Neither model is adopted. Next steps
|
||||
are speech gating/alignment proof, representative negative controls, actual
|
||||
production integration and an equivalent shared-pipeline rerun; the media rating
|
||||
remains 4/10.
|
||||
Reference in new issue
Block a user