Validate complete speech decode windows and retain rejected ASR output

docs/SPEECH-ALIGNMENT.md

@@ -1,14 +1,15 @@
 # Local spoken-passage alignment
 
-Status: researched implementation proposal, 2026-10-13. No transcription worker,
-download integration, or ASR quality result is implemented by this document.
+Status: implementation proposal with an internal CPU diagnostic, 2026-10-13.
+Production transcription, download integration and retrieval acceptance are open.
 Google EmbeddingGemma 2 remains the production semantic model.
 
 A bounded [CPU diagnostic harness](../tools/speech-eval/README.md) is now prepared.
 Seven structural tests passed on Windows, including actual owned subprocess
 failure, timeout, output overflow and retained diagnostics. Both candidate models
-have now been downloaded and their exact bytes/SHA verified. Neither has been
-loaded or evaluated; these tests establish no speech quality.
+have now been downloaded and their exact bytes/SHA verified. Both loaded in the
+internal experiment, whose truncated audio invalidated quality observations.
+Corrected evaluation remains pending; structural tests establish no speech quality.
 
 ## Why add a transcript lane
 
@@ -44,8 +45,8 @@ after checking its bundled upstream revision and backend build flags; avoid mixi
 it blindly with the pinned Google native stack.
 [Rust API](https://docs.rs/whisper-rs/0.16.0/whisper_rs/struct.FullParams.html).
 
-No runtime build or genuine inference proof was run during this research. Obtain
-the coordinator's CPU slot before building or transcribing in the shared VM.
+The initial research performed no inference. Subsequent coordinator-approved CPU
+diagnostics are recorded below; production integration and acceptance remain open.
 
 ## Pinned assets and distribution
 
@@ -64,8 +65,8 @@ Use immutable download URLs of the form
 Both downloads were independently checked against these exact bytes/SHA values
 on 2026-10-13. The clean runtime checkout resolves to the pinned commit; model
 URLs, verified hashes and MIT notices are retained under ignored
-`.searchi/speech/provisioning.json`. This is provisioning evidence only: no
-runtime build, model loading or inference has yet been performed. Commit an
+`.searchi/speech/provisioning.json`. These hashes establish provisioning;
+the subsequent internal runtime experiment does not establish release acceptance. Commit an
 accepted distribution manifest only after model loading and runtime provenance
 checks. Never accept upstream `for-tests-*` empty models.
 [Upstream model provenance](https://github.com/ggml-org/whisper.cpp/blob/master/models/README.md),
@@ -176,3 +177,51 @@ transcripts, worker errors and inferred content never become executable instruct
 Next implementation slice: isolated worker + manifest verification + genuine NASA
 proof. Keep the initial runtime experiment separate from schema/ranking changes
 until it establishes that recognized segments actually fix the measured problem.
+
+## Actual internal CPU diagnostic, 2026-10-13
+
+The clean pinned v1.9.5 source compiled with MSVC in **54.655 seconds**, jobs one,
+static whisper/ggml, `GGML_NATIVE=OFF`; CUDA, Vulkan, Metal, BLAS and OpenMP were
+explicitly disabled in the recorded CMake cache. This is an **internal CPU test
+executable**, not a normal distribution or portability proof. The compiled CPU
+variant uses AVX2; an older-CPU release has not been validated. The executable SHA
+is `12c681b4bcd9c7679b647b9a041e121a2f9e72cb2a9c6f189f529ec8b79c36f8`.
+Help flags were checked and actual runs pass `--no-gpu --threads 1 --processors 1`.
+Dumpbin found Windows system/UCRT and MSVC runtime imports, with no external
+ggml/whisper/GPU imports. Existing VM runtime availability is not a clean-machine
+or older-Windows dependency-closure guarantee.
+
+The initial bounded base experiment completed all five source files; the small
+experiment was stopped by terminating only its owned process tree after a material
+harness bug was identified. Both are classified **INVALID_TRUNCATED_AUDIO**.
+`atrim=end_sample` was applied before explicit 16 kHz resampling: a planned
+30-second window supplied approximately 10 seconds from 48 kHz sources or 10.884
+seconds from 44.1 kHz sources. Recognition omissions and process costs from those
+runs cannot establish complete-window model quality or throughput. Earlier draft
+observations about base/small recognition are withdrawn; no ASR model default or
+improved search score follows from these invalid experiments.
+
+Actual downloaded NASA publisher captions are independently retained: 2204 bytes,
+SHA `80d82def36547d95c512034deb9df9d3d0160579a43c8177e6ca988d3b02b04b`.
+Their utterance intervals agree with the frozen v1 labels. Comparing recognized
+text against these captions must be repeated after corrected decoding.
+[NASA publisher captions](https://images-assets.nasa.gov/video/Artemis%20Origins/Artemis%20Origins.srt).
+No independent human audio listening has been performed.
+
+The decoder now explicitly resamples to 16000 Hz before trimming and resets sample
+timestamps. Every decoded WAV records its frame count and rejects coverage outside
+expected duration minus at most one millisecond. Actual CPU FFmpeg regressions
+verified both 44.1 kHz and 48 kHz sources: a 30-second window has exactly 480000
+samples and a one-second tail exactly 16000. Generated silent WAVs exercise
+resampling mechanics only; they are not ASR quality fixtures. Ten tests passed in
+1.406 seconds, including rejection of threefold truncation, bounded owned-child
+errors/timeouts/output, retained diagnostics and continuation after invalid ASR
+timestamp output. They do not establish speech recognition quality.
+
+Historical raw/checkpoint hashes and invalid classification remain under ignored
+`.searchi/speech/invalid-truncated-audio.json`; model bytes and actual raw outputs
+are excluded from Git. The owned sequential session 20331 is terminal with exit 1
+following intentional invalid-experiment termination. Corrected base/small
+inference is paused pending the coordinator's CPU slot, followed by independent
+caption review and genuine Google transcript ranking. Neither candidate is an
+accepted product default; the media quality rating remains 4/10.

tools/speech-eval/README.md

@@ -26,7 +26,11 @@ does not implement overlap/dedup or production budgets. Segment JSON offsets are
 already milliseconds and get the window offset added exactly once.
 
 Each completed window writes an incomplete checkpoint; the final report marks
-complete. Errors retain bounded owned logs outside temporary scratch, which cleans
+complete evaluation, with `quality_failed` and explicit `validation_failures` for
+invalid ASR timestamps. Bounded raw ASR JSON remains in each checkpoint before
+validity is decided; invalid times are never clamped into evidence, and later
+windows continue. The final report records the executed harness SHA. Process
+errors retain bounded owned logs and exit/time diagnostics outside temporary scratch, which cleans
 decoded WAV/JSON on exit. Cancellation terminates only owned subprocess trees.
 Limits are supervisory checks, not OS resource sandbox guarantees. Recognized text
 is private local diagnostic data; never commit raw logs/transcripts by default.

tools/speech-eval/run.py

@@ -97,6 +97,36 @@ def segments(document, offset_ms, duration_ms):
     return parsed
 
 
+def recognized_window(raw, offset, length, decode_ms, asr_ms):
+    report = {"offset_ms": offset, "duration_ms": length, "decode_ms": decode_ms,
+              "asr_process_ms": asr_ms, "actual_whisper_json": raw}
+    try:
+        report["segments"] = segments(raw, offset, length)
+    except (ValueError, KeyError, TypeError) as error:
+        report["segments"] = []
+        report["validation_error"] = str(error)
+    return report
+
+
+def decode_command(ffmpeg, source, offset, length, wav):
+    return [str(ffmpeg.resolve()), "-nostdin", "-hide_banner", "-v", "error",
+            "-protocol_whitelist", "file,pipe", "-threads", "1", "-ss", str(offset / 1000), "-t", str(length / 1000),
+            "-i", str(source.resolve()), "-map", "0:a:0", "-vn", "-ac", "1", "-ar", "16000",
+            "-af", f"aresample=16000,atrim=end_sample={length * 16},asetpts=PTS-STARTPTS",
+            "-c:a", "pcm_s16le", "-threads", "1", "-y", str(wav)]
+
+
+def decoded_frames(wav, length):
+    with wave.open(str(wav), "rb") as audio:
+        if audio.getframerate() != 16000 or audio.getnchannels() != 1 or audio.getsampwidth() != 2:
+            raise ValueError("Invalid decoded PCM")
+        frames = audio.getnframes()
+    # Container duration rounding allows at most one millisecond's shortfall.
+    if not max(1, length * 16 - 16) <= frames <= length * 16:
+        raise ValueError(f"Decoded sample coverage mismatch: expected {length * 16}, got {frames}")
+    return frames
+
+
 def main():
     parser = argparse.ArgumentParser(description=__doc__)
     parser.add_argument("--whisper", type=Path, required=True)
@@ -145,20 +175,16 @@ def main():
                 prefix = scratch / "recognized"
                 output = prefix.with_suffix(".json")
                 output.unlink(missing_ok=True)
-                decode_ms = run_owned([str(args.ffmpeg.resolve()), "-nostdin", "-hide_banner", "-v", "error",
-                    "-protocol_whitelist", "file,pipe", "-threads", "1", "-ss", str(offset / 1000), "-t", str(length / 1000),
-                    "-i", str(source.resolve()), "-map", "0:a:0", "-vn", "-ac", "1", "-ar", "16000",
-                    "-af", f"atrim=end_sample={length * 16}", "-c:a", "pcm_s16le", "-threads", "1", "-y", str(wav)],
+                decode_ms = run_owned(decode_command(args.ffmpeg, source, offset, length, wav),
                     scratch, "decode", [(wav, 970000)], 60)
-                with wave.open(str(wav), "rb") as audio:
-                    if audio.getframerate() != 16000 or audio.getnchannels() != 1 or audio.getsampwidth() != 2:
-                        raise ValueError("Invalid decoded PCM")
+                frames = decoded_frames(wav, length)
                 asr_ms = run_owned([str(args.whisper.resolve()), "--no-gpu", "--threads", "1", "--processors", "1",
                     "--language", "auto", "--model", str(args.model.resolve()), "--file", str(wav),
                     "--output-json", "--output-file", str(prefix)], scratch, "asr", [(output, MAX_OUTPUT)], 600)
                 raw = json.loads(output.read_text(encoding="utf-8"))
-                result["windows"].append({"offset_ms": offset, "duration_ms": length, "decode_ms": decode_ms,
-                    "asr_process_ms": asr_ms, "segments": segments(raw, offset, length), "actual_whisper_json": raw})
+                # Preserve genuine invalid output, never clamp it into valid evidence.
+                result["windows"].append(recognized_window(raw, offset, length, decode_ms, asr_ms))
+                result["windows"][-1]["decoded_frames"] = frames
                 args.output.parent.mkdir(parents=True, exist_ok=True)
                 checkpoint = {"complete": False, "fixture_sha256": sha256(args.fixture),
                               "model_sha256": digest, "worker_sha256": args.whisper_sha256,
@@ -167,12 +193,16 @@ def main():
             if source.stat().st_size != asset["bytes"] or sha256(source) != asset["sha256"]:
                 raise ValueError("Source changed during recognition")
             rows.append(result)
-    report = {"version": 1, "complete": True, "scope": "Actual ASR diagnostic, not Google retrieval or production integration",
+    failures = [{"asset": row["asset"], "offset_ms": window["offset_ms"], "error": window["validation_error"]}
+                for row in rows for window in row["windows"] if "validation_error" in window]
+    report = {"version": 1, "complete": True, "quality_failed": bool(failures), "validation_failures": failures,
+              "scope": "Actual ASR diagnostic, not Google retrieval or production integration",
               "fixture_sha256": sha256(args.fixture), "upstream_commit": UPSTREAM,
               "worker_sha256": args.whisper_sha256, "model_sha256": digest, "candidate": args.candidate,
               "ffmpeg_sha256": sha256(args.ffmpeg), "ffprobe_sha256": sha256(args.ffprobe),
               "backend": "forced CPU", "threads": 1,
-              "policy": "nonoverlapping30s;independent-process-per-window;auto-language;segment-times",
+              "harness_sha256": sha256(Path(__file__)),
+              "policy": "aresample16000-before-atrim-v2;samplecoverage-1ms;nonoverlapping30s;independent-process-per-window;auto-language;segment-times",
               "rows": rows}
     args.output.parent.mkdir(parents=True, exist_ok=True)
     args.output.write_text(json.dumps(report, indent=2, ensure_ascii=False) + "\n", encoding="utf-8")

tools/speech-eval/test_contract.py

@@ -4,11 +4,53 @@ from pathlib import Path
 import unittest
 import tempfile
 import sys
-from run import segments, run_owned
+import os
+import wave
+from run import segments, run_owned, recognized_window, decode_command, decoded_frames
 from score import score
 
 
 class ContractTests(unittest.TestCase):
+    @unittest.skipUnless(os.environ.get("SEARCHI_TEST_FFMPEG"), "Requires verified CPU FFmpeg")
+    def test_actual_ffmpeg_full_window_and_partial_tail_sample_coverage(self):
+        ffmpeg = Path(os.environ["SEARCHI_TEST_FFMPEG"])
+        with tempfile.TemporaryDirectory() as parent:
+            scratch = Path(parent)
+            for rate in [44100, 48000]:
+                with self.subTest(rate=rate):
+                    source = scratch / f"source-{rate}.wav"
+                    with wave.open(str(source), "wb") as audio:
+                        audio.setnchannels(1)
+                        audio.setsampwidth(2)
+                        audio.setframerate(rate)
+                        audio.writeframes(b"\x00\x00" * (rate * 31))
+                    for offset, length, expected in [(0, 30000, 480000), (30000, 1000, 16000)]:
+                        output = scratch / "decoded.wav"
+                        run_owned(decode_command(ffmpeg, source, offset, length, output), scratch,
+                                  "samplecoverage", [(output, 970000)], 30)
+                        self.assertEqual(decoded_frames(output, length), expected)
+
+    def test_threefold_truncation_rejected(self):
+        with tempfile.TemporaryDirectory() as parent:
+            path = Path(parent) / "truncated.wav"
+            with wave.open(str(path), "wb") as audio:
+                audio.setnchannels(1)
+                audio.setsampwidth(2)
+                audio.setframerate(16000)
+                audio.writeframes(b"\x00\x00" * 160000)
+            with self.assertRaises(ValueError):
+                decoded_frames(path, 30000)
+
+    def test_invalid_actual_json_is_retained_and_next_window_parses(self):
+        invalid = {"transcription": [{"offsets": {"from": 0, "to": 30010}, "text": "synthetic test"}]}
+        valid = {"transcription": [{"offsets": {"from": 0, "to": 10}, "text": "synthetic test"}]}
+        rows = [recognized_window(raw, offset, 30000, 1, 2) for raw, offset in [(invalid, 0), (valid, 30000)]]
+        self.assertEqual(rows[0]["actual_whisper_json"], invalid)
+        self.assertEqual(rows[0]["segments"], [])
+        self.assertIn("validation_error", rows[0])
+        self.assertNotIn("validation_error", rows[1])
+        self.assertEqual(rows[1]["segments"][0]["start_ms"], 30000)
+
     def test_timeout_and_output_overflow_terminate_owned_child(self):
         for name, code, error, timeout in [
             ("timeout", "import time; time.sleep(10)", TimeoutError, 0.1),