Freeze unseen music stress corpus and independently verified gated DTW experiment

docs/SPEECH-DTW.md

@@ -0,0 +1,88 @@
+# DTW diagnostic: strict UTF-8 failure
+
+Status: internal CPU diagnosis, 2026-10-13. No accepted speech/retrieval score,
+production ASR integration or new media rating follows from this evidence.
+Google EmbeddingGemma 2 remains the production semantic model.
+
+The original fifteen-window DTW experiment stopped on strict UTF-8 parsing.
+Its temporary raw output was cleaned, so that first run's exact bytes could not
+be inspected. Its partial report remains unchanged. A separate coordinator-
+authorized diagnostic reproduced exactly the second `music01` window: source
+offset 30000 ms, duration 30000 ms, primary audio stream, 480000 decoded samples.
+No labels selected text/timestamps, and the whole experiment was not rerun.
+
+## Revalidated execution
+
+The whisper.cpp checkout was clean at pinned commit
+`d1be6fde11ac6e0407606b4e42fe72d34add8037`. Before execution, the fixture and
+source hashes, base model size/hash, worker hash and both decoder hashes were
+checked against frozen evidence. The owned runner retained raw output, WAV,
+probe, stderr and report directly under ignored
+`.searchi/speech/dtw-window-diagnostic-v1`; no temporary cleanup removed them.
+
+| Artifact | SHA-256 |
+| --- | --- |
+| Frozen fixture | `b4e465a0eb9582b826eebd2a20797e1680e4907e336d02d0ef2194e2ba0af393` |
+| Source music01 | `29fcd20414ee19c2a5aa4339eaf6375079dc87959051ab0aea32c76df52f34dc` |
+| CPU worker | `12c681b4bcd9c7679b647b9a041e121a2f9e72cb2a9c6f189f529ec8b79c36f8` |
+| Multilingual base model | `60ed5bc3dd14eea856493d334349b405782ddcaf0028d4b5df4088345fba2efe` |
+| FFmpeg | `b778ca5786ee4b0b4f37f34478617093d2fdb9a532d281fb76cda69d1d579d46` |
+| FFprobe | `4e1b0afd27bf02d79739f27b2d650137470925afb19d54b5e89a708b7a389236` |
+| Raw JSON, 29148 bytes | `f1de2c295b0c9a223619abe6dbf34823be6a6aa3dbcad69fea1d6121fa0c200e` |
+| Native log | `28d174198a3ef980aa5c8c1237618f450a19b0a80c8a270209dce113a8de3a7a` |
+| Diagnostic report | `b4b0dedfac6365c59793806801b1fc5d9f8f203891b623fb86ad508352ad0114` |
+
+Windows VM execution forced CPU: `SEARCHI_FORCE_CPU=1`, native flags
+`--no-gpu --threads 1 --processors 1 --language auto --output-json-full
+--dtw base --no-flash-attn`, verified base model and the decoded owned WAV.
+The existing bounded decoder used absolute 30-second seeking and sample trimming;
+its output contained exactly 480000 samples. Native ASR exited zero after
+42493.25 ms; decoding took 271.42 ms. The owned diagnostic runner exited zero
+after recording `strict_parse: rejected`. This is not a successful ASR result
+or a warmed throughput measurement. No other model inference ran concurrently
+under this coordinator's CPU slot; unrelated VM load was not instrumented.
+
+## Exact failure and pinned native source
+
+Strict UTF-8 decoding rejects bytes **1137–1138** (zero-based half-open range
+1137–1139). The segment text ends with bytes `E2 80` immediately before the
+closing JSON quote: a three-byte UTF-8 sequence has no final continuation byte.
+The last ordinary token, id 913, also ends with exactly `E2 80` at byte 28900.
+Earlier token text contains complete `E2 80 8D` codepoints. Raw bytes were
+inspected directly, without replacement decoding or writing repaired JSON.
+
+The pinned native core constructs segment text by concatenating token strings.
+The CLI full-JSON serializer merges adjacent token byte pieces while a codepoint
+needs continuation **and another token exists**. At the last token there is no
+continuation to merge; it writes the remaining incomplete bytes. Its segment
+text serializer likewise writes the already incomplete segment bytes. The
+escaping function only escapes quotes/backslashes; it cannot restore a missing
+codepoint byte. Thus this is incomplete recognized byte output, reproduced in
+both segment and token text, rather than a Python ANSI conversion or DTW time
+unit error. The exact internal reason the decoder terminated this token sequence
+has not been instrumented and remains unproven.
+
+Relevant pinned source: [CLI serializer and token merging](https://github.com/ggml-org/whisper.cpp/blob/d1be6fde11ac6e0407606b4e42fe72d34add8037/examples/cli/cli.cpp),
+[UTF-8 continuation helper](https://github.com/ggml-org/whisper.cpp/blob/d1be6fde11ac6e0407606b4e42fe72d34add8037/examples/common-whisper.cpp),
+and [segment construction](https://github.com/ggml-org/whisper.cpp/blob/d1be6fde11ac6e0407606b4e42fe72d34add8037/src/whisper.cpp).
+
+## Minimal next change and limits
+
+Before changing a native worker, add a strict UTF-8 validator at its JSON boundary
+for original segment strings and merged token strings. If original recognized
+bytes remain invalid after adjacent-token merging, return an explicit typed
+recognition-output error with bounded raw evidence; never replace characters,
+truncate text, synthesize missing bytes or fabricate timestamps. Test a valid
+split codepoint and an incomplete final codepoint. This makes the failure
+machine-readable; it does **not** repair recognition or establish improved
+retrieval. An alternative timestamp/recognition policy must be declared and
+evaluated separately before any default changes.
+
+The Python harness already fails closed, retains bounded raw bytes with SHA and
+parse positions, and atomically records an incomplete, ineligible checkpoint.
+The original partial DTW report contains three NASA windows with 21 valid
+point-based sentence spans and one separately retained ordinary-interval error.
+It lacks remaining piano windows and three other competitors. Neither that
+partial report nor this one-window diagnosis can receive an accepted quality
+score. Packaging, independent bilingual speech, silence/music hallucination,
+resource peaks and actual production hybrid integration remain open.