# DTW diagnostic: strict UTF-8 failure

Status: internal CPU diagnosis, 2026-10-13. No accepted speech/retrieval score,
production ASR integration or new media rating follows from this evidence.
Google EmbeddingGemma 2 remains the production semantic model.

The original fifteen-window DTW experiment stopped on strict UTF-8 parsing.
Its temporary raw output was cleaned, so that first run's exact bytes could not
be inspected. Its partial report remains unchanged. A separate coordinator-
authorized diagnostic reproduced exactly the second `music01` window: source
offset 30000 ms, duration 30000 ms, primary audio stream, 480000 decoded samples.
No labels selected text/timestamps, and the whole experiment was not rerun.

## Revalidated execution

The whisper.cpp checkout was clean at pinned commit
`d1be6fde11ac6e0407606b4e42fe72d34add8037`. Before execution, the fixture and
source hashes, base model size/hash, worker hash and both decoder hashes were
checked against frozen evidence. The owned runner retained raw output, WAV,
probe, stderr and report directly under ignored
`.searchi/speech/dtw-window-diagnostic-v1`; no temporary cleanup removed them.

| Artifact | SHA-256 |
| --- | --- |
| Frozen fixture | `b4e465a0eb9582b826eebd2a20797e1680e4907e336d02d0ef2194e2ba0af393` |
| Source music01 | `29fcd20414ee19c2a5aa4339eaf6375079dc87959051ab0aea32c76df52f34dc` |
| CPU worker | `12c681b4bcd9c7679b647b9a041e121a2f9e72cb2a9c6f189f529ec8b79c36f8` |
| Multilingual base model | `60ed5bc3dd14eea856493d334349b405782ddcaf0028d4b5df4088345fba2efe` |
| FFmpeg | `b778ca5786ee4b0b4f37f34478617093d2fdb9a532d281fb76cda69d1d579d46` |
| FFprobe | `4e1b0afd27bf02d79739f27b2d650137470925afb19d54b5e89a708b7a389236` |
| Raw JSON, 29148 bytes | `f1de2c295b0c9a223619abe6dbf34823be6a6aa3dbcad69fea1d6121fa0c200e` |
| Native log | `28d174198a3ef980aa5c8c1237618f450a19b0a80c8a270209dce113a8de3a7a` |
| Diagnostic report | `b4b0dedfac6365c59793806801b1fc5d9f8f203891b623fb86ad508352ad0114` |

Windows VM execution forced CPU: `SEARCHI_FORCE_CPU=1`, native flags
`--no-gpu --threads 1 --processors 1 --language auto --output-json-full
--dtw base --no-flash-attn`, verified base model and the decoded owned WAV.
The existing bounded decoder used absolute 30-second seeking and sample trimming;
its output contained exactly 480000 samples. Native ASR exited zero after
42493.25 ms; decoding took 271.42 ms. The owned diagnostic runner exited zero
after recording `strict_parse: rejected`. This is not a successful ASR result
or a warmed throughput measurement. No other model inference ran concurrently
under this coordinator's CPU slot; unrelated VM load was not instrumented.

## Exact failure and pinned native source

Strict UTF-8 decoding rejects bytes **1137–1138** (zero-based half-open range
1137–1139). The segment text ends with bytes `E2 80` immediately before the
closing JSON quote: a three-byte UTF-8 sequence has no final continuation byte.
The last ordinary token, id 913, also ends with exactly `E2 80` at byte 28900.
Earlier token text contains complete `E2 80 8D` codepoints. Raw bytes were
inspected directly, without replacement decoding or writing repaired JSON.

The pinned native core constructs segment text by concatenating token strings.
The CLI full-JSON serializer merges adjacent token byte pieces while a codepoint
needs continuation **and another token exists**. At the last token there is no
continuation to merge; it writes the remaining incomplete bytes. Its segment
text serializer likewise writes the already incomplete segment bytes. The
escaping function only escapes quotes/backslashes; it cannot restore a missing
codepoint byte. Thus this is incomplete recognized byte output, reproduced in
both segment and token text, rather than a Python ANSI conversion or DTW time
unit error. The exact internal reason the decoder terminated this token sequence
has not been instrumented and remains unproven.

Relevant pinned source: [CLI serializer and token merging](https://github.com/ggml-org/whisper.cpp/blob/d1be6fde11ac6e0407606b4e42fe72d34add8037/examples/cli/cli.cpp),
[UTF-8 continuation helper](https://github.com/ggml-org/whisper.cpp/blob/d1be6fde11ac6e0407606b4e42fe72d34add8037/examples/common-whisper.cpp),
and [segment construction](https://github.com/ggml-org/whisper.cpp/blob/d1be6fde11ac6e0407606b4e42fe72d34add8037/src/whisper.cpp).

## Minimal next change and limits

Before changing a native worker, add a strict UTF-8 validator at its JSON boundary
for original segment strings and merged token strings. If original recognized
bytes remain invalid after adjacent-token merging, return an explicit typed
recognition-output error with bounded raw evidence; never replace characters,
truncate text, synthesize missing bytes or fabricate timestamps. Test a valid
split codepoint and an incomplete final codepoint. This makes the failure
machine-readable; it does **not** repair recognition or establish improved
retrieval. An alternative timestamp/recognition policy must be declared and
evaluated separately before any default changes.

The Python harness already fails closed, retains bounded raw bytes with SHA and
parse positions, and atomically records an incomplete, ineligible checkpoint.
The original partial DTW report contains three NASA windows with 21 valid
point-based sentence spans and one separately retained ordinary-interval error.
It lacks remaining piano windows and three other competitors. Neither that
partial report nor this one-window diagnosis can receive an accepted quality
score. Packaging, independent bilingual speech, silence/music hallucination,
resource peaks and actual production hybrid integration remain open.