5.6 KiB
DTW diagnostic: strict UTF-8 failure
Status: internal CPU diagnosis, 2026-10-13. No accepted speech/retrieval score, production ASR integration or new media rating follows from this evidence. Google EmbeddingGemma 2 remains the production semantic model.
The original fifteen-window DTW experiment stopped on strict UTF-8 parsing.
Its temporary raw output was cleaned, so that first run's exact bytes could not
be inspected. Its partial report remains unchanged. A separate coordinator-
authorized diagnostic reproduced exactly the second music01 window: source
offset 30000 ms, duration 30000 ms, primary audio stream, 480000 decoded samples.
No labels selected text/timestamps, and the whole experiment was not rerun.
Revalidated execution
The whisper.cpp checkout was clean at pinned commit
d1be6fde11ac6e0407606b4e42fe72d34add8037. Before execution, the fixture and
source hashes, base model size/hash, worker hash and both decoder hashes were
checked against frozen evidence. The owned runner retained raw output, WAV,
probe, stderr and report directly under ignored
.searchi/speech/dtw-window-diagnostic-v1; no temporary cleanup removed them.
| Artifact | SHA-256 |
|---|---|
| Frozen fixture | b4e465a0eb9582b826eebd2a20797e1680e4907e336d02d0ef2194e2ba0af393 |
| Source music01 | 29fcd20414ee19c2a5aa4339eaf6375079dc87959051ab0aea32c76df52f34dc |
| CPU worker | 12c681b4bcd9c7679b647b9a041e121a2f9e72cb2a9c6f189f529ec8b79c36f8 |
| Multilingual base model | 60ed5bc3dd14eea856493d334349b405782ddcaf0028d4b5df4088345fba2efe |
| FFmpeg | b778ca5786ee4b0b4f37f34478617093d2fdb9a532d281fb76cda69d1d579d46 |
| FFprobe | 4e1b0afd27bf02d79739f27b2d650137470925afb19d54b5e89a708b7a389236 |
| Raw JSON, 29148 bytes | f1de2c295b0c9a223619abe6dbf34823be6a6aa3dbcad69fea1d6121fa0c200e |
| Native log | 28d174198a3ef980aa5c8c1237618f450a19b0a80c8a270209dce113a8de3a7a |
| Diagnostic report | b4b0dedfac6365c59793806801b1fc5d9f8f203891b623fb86ad508352ad0114 |
Windows VM execution forced CPU: SEARCHI_FORCE_CPU=1, native flags
--no-gpu --threads 1 --processors 1 --language auto --output-json-full --dtw base --no-flash-attn, verified base model and the decoded owned WAV.
The existing bounded decoder used absolute 30-second seeking and sample trimming;
its output contained exactly 480000 samples. Native ASR exited zero after
42493.25 ms; decoding took 271.42 ms. The owned diagnostic runner exited zero
after recording strict_parse: rejected. This is not a successful ASR result
or a warmed throughput measurement. No other model inference ran concurrently
under this coordinator's CPU slot; unrelated VM load was not instrumented.
Exact failure and pinned native source
Strict UTF-8 decoding rejects bytes 1137–1138 (zero-based half-open range
1137–1139). The segment text ends with bytes E2 80 immediately before the
closing JSON quote: a three-byte UTF-8 sequence has no final continuation byte.
The last ordinary token, id 913, also ends with exactly E2 80 at byte 28900.
Earlier token text contains complete E2 80 8D codepoints. Raw bytes were
inspected directly, without replacement decoding or writing repaired JSON.
The pinned native core constructs segment text by concatenating token strings. The CLI full-JSON serializer merges adjacent token byte pieces while a codepoint needs continuation and another token exists. At the last token there is no continuation to merge; it writes the remaining incomplete bytes. Its segment text serializer likewise writes the already incomplete segment bytes. The escaping function only escapes quotes/backslashes; it cannot restore a missing codepoint byte. Thus this is incomplete recognized byte output, reproduced in both segment and token text, rather than a Python ANSI conversion or DTW time unit error. The exact internal reason the decoder terminated this token sequence has not been instrumented and remains unproven.
Relevant pinned source: CLI serializer and token merging, UTF-8 continuation helper, and segment construction.
Minimal next change and limits
Before changing a native worker, add a strict UTF-8 validator at its JSON boundary for original segment strings and merged token strings. If original recognized bytes remain invalid after adjacent-token merging, return an explicit typed recognition-output error with bounded raw evidence; never replace characters, truncate text, synthesize missing bytes or fabricate timestamps. Test a valid split codepoint and an incomplete final codepoint. This makes the failure machine-readable; it does not repair recognition or establish improved retrieval. An alternative timestamp/recognition policy must be declared and evaluated separately before any default changes.
The Python harness already fails closed, retains bounded raw bytes with SHA and parse positions, and atomically records an incomplete, ineligible checkpoint. The original partial DTW report contains three NASA windows with 21 valid point-based sentence spans and one separately retained ordinary-interval error. It lacks remaining piano windows and three other competitors. Neither that partial report nor this one-window diagnosis can receive an accepted quality score. Packaging, independent bilingual speech, silence/music hallucination, resource peaks and actual production hybrid integration remain open.