diff --git a/docs/SPEECH-ALIGNMENT.md b/docs/SPEECH-ALIGNMENT.md index 7b25b4c..fffd097 100644 --- a/docs/SPEECH-ALIGNMENT.md +++ b/docs/SPEECH-ALIGNMENT.md @@ -1,14 +1,15 @@ # Local spoken-passage alignment -Status: researched implementation proposal, 2026-10-13. No transcription worker, -download integration, or ASR quality result is implemented by this document. +Status: implementation proposal with an internal CPU diagnostic, 2026-10-13. +Production transcription, download integration and retrieval acceptance are open. Google EmbeddingGemma 2 remains the production semantic model. A bounded [CPU diagnostic harness](../tools/speech-eval/README.md) is now prepared. Seven structural tests passed on Windows, including actual owned subprocess failure, timeout, output overflow and retained diagnostics. Both candidate models -have now been downloaded and their exact bytes/SHA verified. Neither has been -loaded or evaluated; these tests establish no speech quality. +have now been downloaded and their exact bytes/SHA verified. Both loaded in the +internal experiment, whose truncated audio invalidated quality observations. +Corrected evaluation remains pending; structural tests establish no speech quality. ## Why add a transcript lane @@ -44,8 +45,8 @@ after checking its bundled upstream revision and backend build flags; avoid mixi it blindly with the pinned Google native stack. [Rust API](https://docs.rs/whisper-rs/0.16.0/whisper_rs/struct.FullParams.html). -No runtime build or genuine inference proof was run during this research. Obtain -the coordinator's CPU slot before building or transcribing in the shared VM. +The initial research performed no inference. Subsequent coordinator-approved CPU +diagnostics are recorded below; production integration and acceptance remain open. ## Pinned assets and distribution @@ -64,8 +65,8 @@ Use immutable download URLs of the form Both downloads were independently checked against these exact bytes/SHA values on 2026-10-13. The clean runtime checkout resolves to the pinned commit; model URLs, verified hashes and MIT notices are retained under ignored -`.searchi/speech/provisioning.json`. This is provisioning evidence only: no -runtime build, model loading or inference has yet been performed. Commit an +`.searchi/speech/provisioning.json`. These hashes establish provisioning; +the subsequent internal runtime experiment does not establish release acceptance. Commit an accepted distribution manifest only after model loading and runtime provenance checks. Never accept upstream `for-tests-*` empty models. [Upstream model provenance](https://github.com/ggml-org/whisper.cpp/blob/master/models/README.md), @@ -176,3 +177,51 @@ transcripts, worker errors and inferred content never become executable instruct Next implementation slice: isolated worker + manifest verification + genuine NASA proof. Keep the initial runtime experiment separate from schema/ranking changes until it establishes that recognized segments actually fix the measured problem. + +## Actual internal CPU diagnostic, 2026-10-13 + +The clean pinned v1.9.5 source compiled with MSVC in **54.655 seconds**, jobs one, +static whisper/ggml, `GGML_NATIVE=OFF`; CUDA, Vulkan, Metal, BLAS and OpenMP were +explicitly disabled in the recorded CMake cache. This is an **internal CPU test +executable**, not a normal distribution or portability proof. The compiled CPU +variant uses AVX2; an older-CPU release has not been validated. The executable SHA +is `12c681b4bcd9c7679b647b9a041e121a2f9e72cb2a9c6f189f529ec8b79c36f8`. +Help flags were checked and actual runs pass `--no-gpu --threads 1 --processors 1`. +Dumpbin found Windows system/UCRT and MSVC runtime imports, with no external +ggml/whisper/GPU imports. Existing VM runtime availability is not a clean-machine +or older-Windows dependency-closure guarantee. + +The initial bounded base experiment completed all five source files; the small +experiment was stopped by terminating only its owned process tree after a material +harness bug was identified. Both are classified **INVALID_TRUNCATED_AUDIO**. +`atrim=end_sample` was applied before explicit 16 kHz resampling: a planned +30-second window supplied approximately 10 seconds from 48 kHz sources or 10.884 +seconds from 44.1 kHz sources. Recognition omissions and process costs from those +runs cannot establish complete-window model quality or throughput. Earlier draft +observations about base/small recognition are withdrawn; no ASR model default or +improved search score follows from these invalid experiments. + +Actual downloaded NASA publisher captions are independently retained: 2204 bytes, +SHA `80d82def36547d95c512034deb9df9d3d0160579a43c8177e6ca988d3b02b04b`. +Their utterance intervals agree with the frozen v1 labels. Comparing recognized +text against these captions must be repeated after corrected decoding. +[NASA publisher captions](https://images-assets.nasa.gov/video/Artemis%20Origins/Artemis%20Origins.srt). +No independent human audio listening has been performed. + +The decoder now explicitly resamples to 16000 Hz before trimming and resets sample +timestamps. Every decoded WAV records its frame count and rejects coverage outside +expected duration minus at most one millisecond. Actual CPU FFmpeg regressions +verified both 44.1 kHz and 48 kHz sources: a 30-second window has exactly 480000 +samples and a one-second tail exactly 16000. Generated silent WAVs exercise +resampling mechanics only; they are not ASR quality fixtures. Ten tests passed in +1.406 seconds, including rejection of threefold truncation, bounded owned-child +errors/timeouts/output, retained diagnostics and continuation after invalid ASR +timestamp output. They do not establish speech recognition quality. + +Historical raw/checkpoint hashes and invalid classification remain under ignored +`.searchi/speech/invalid-truncated-audio.json`; model bytes and actual raw outputs +are excluded from Git. The owned sequential session 20331 is terminal with exit 1 +following intentional invalid-experiment termination. Corrected base/small +inference is paused pending the coordinator's CPU slot, followed by independent +caption review and genuine Google transcript ranking. Neither candidate is an +accepted product default; the media quality rating remains 4/10. diff --git a/tools/speech-eval/README.md b/tools/speech-eval/README.md index 678ce3e..5f8c9ca 100644 --- a/tools/speech-eval/README.md +++ b/tools/speech-eval/README.md @@ -26,7 +26,11 @@ does not implement overlap/dedup or production budgets. Segment JSON offsets are already milliseconds and get the window offset added exactly once. Each completed window writes an incomplete checkpoint; the final report marks -complete. Errors retain bounded owned logs outside temporary scratch, which cleans +complete evaluation, with `quality_failed` and explicit `validation_failures` for +invalid ASR timestamps. Bounded raw ASR JSON remains in each checkpoint before +validity is decided; invalid times are never clamped into evidence, and later +windows continue. The final report records the executed harness SHA. Process +errors retain bounded owned logs and exit/time diagnostics outside temporary scratch, which cleans decoded WAV/JSON on exit. Cancellation terminates only owned subprocess trees. Limits are supervisory checks, not OS resource sandbox guarantees. Recognized text is private local diagnostic data; never commit raw logs/transcripts by default. diff --git a/tools/speech-eval/run.py b/tools/speech-eval/run.py index 57ace6f..830b21e 100644 --- a/tools/speech-eval/run.py +++ b/tools/speech-eval/run.py @@ -97,6 +97,36 @@ def segments(document, offset_ms, duration_ms): return parsed +def recognized_window(raw, offset, length, decode_ms, asr_ms): + report = {"offset_ms": offset, "duration_ms": length, "decode_ms": decode_ms, + "asr_process_ms": asr_ms, "actual_whisper_json": raw} + try: + report["segments"] = segments(raw, offset, length) + except (ValueError, KeyError, TypeError) as error: + report["segments"] = [] + report["validation_error"] = str(error) + return report + + +def decode_command(ffmpeg, source, offset, length, wav): + return [str(ffmpeg.resolve()), "-nostdin", "-hide_banner", "-v", "error", + "-protocol_whitelist", "file,pipe", "-threads", "1", "-ss", str(offset / 1000), "-t", str(length / 1000), + "-i", str(source.resolve()), "-map", "0:a:0", "-vn", "-ac", "1", "-ar", "16000", + "-af", f"aresample=16000,atrim=end_sample={length * 16},asetpts=PTS-STARTPTS", + "-c:a", "pcm_s16le", "-threads", "1", "-y", str(wav)] + + +def decoded_frames(wav, length): + with wave.open(str(wav), "rb") as audio: + if audio.getframerate() != 16000 or audio.getnchannels() != 1 or audio.getsampwidth() != 2: + raise ValueError("Invalid decoded PCM") + frames = audio.getnframes() + # Container duration rounding allows at most one millisecond's shortfall. + if not max(1, length * 16 - 16) <= frames <= length * 16: + raise ValueError(f"Decoded sample coverage mismatch: expected {length * 16}, got {frames}") + return frames + + def main(): parser = argparse.ArgumentParser(description=__doc__) parser.add_argument("--whisper", type=Path, required=True) @@ -145,20 +175,16 @@ def main(): prefix = scratch / "recognized" output = prefix.with_suffix(".json") output.unlink(missing_ok=True) - decode_ms = run_owned([str(args.ffmpeg.resolve()), "-nostdin", "-hide_banner", "-v", "error", - "-protocol_whitelist", "file,pipe", "-threads", "1", "-ss", str(offset / 1000), "-t", str(length / 1000), - "-i", str(source.resolve()), "-map", "0:a:0", "-vn", "-ac", "1", "-ar", "16000", - "-af", f"atrim=end_sample={length * 16}", "-c:a", "pcm_s16le", "-threads", "1", "-y", str(wav)], + decode_ms = run_owned(decode_command(args.ffmpeg, source, offset, length, wav), scratch, "decode", [(wav, 970000)], 60) - with wave.open(str(wav), "rb") as audio: - if audio.getframerate() != 16000 or audio.getnchannels() != 1 or audio.getsampwidth() != 2: - raise ValueError("Invalid decoded PCM") + frames = decoded_frames(wav, length) asr_ms = run_owned([str(args.whisper.resolve()), "--no-gpu", "--threads", "1", "--processors", "1", "--language", "auto", "--model", str(args.model.resolve()), "--file", str(wav), "--output-json", "--output-file", str(prefix)], scratch, "asr", [(output, MAX_OUTPUT)], 600) raw = json.loads(output.read_text(encoding="utf-8")) - result["windows"].append({"offset_ms": offset, "duration_ms": length, "decode_ms": decode_ms, - "asr_process_ms": asr_ms, "segments": segments(raw, offset, length), "actual_whisper_json": raw}) + # Preserve genuine invalid output, never clamp it into valid evidence. + result["windows"].append(recognized_window(raw, offset, length, decode_ms, asr_ms)) + result["windows"][-1]["decoded_frames"] = frames args.output.parent.mkdir(parents=True, exist_ok=True) checkpoint = {"complete": False, "fixture_sha256": sha256(args.fixture), "model_sha256": digest, "worker_sha256": args.whisper_sha256, @@ -167,12 +193,16 @@ def main(): if source.stat().st_size != asset["bytes"] or sha256(source) != asset["sha256"]: raise ValueError("Source changed during recognition") rows.append(result) - report = {"version": 1, "complete": True, "scope": "Actual ASR diagnostic, not Google retrieval or production integration", + failures = [{"asset": row["asset"], "offset_ms": window["offset_ms"], "error": window["validation_error"]} + for row in rows for window in row["windows"] if "validation_error" in window] + report = {"version": 1, "complete": True, "quality_failed": bool(failures), "validation_failures": failures, + "scope": "Actual ASR diagnostic, not Google retrieval or production integration", "fixture_sha256": sha256(args.fixture), "upstream_commit": UPSTREAM, "worker_sha256": args.whisper_sha256, "model_sha256": digest, "candidate": args.candidate, "ffmpeg_sha256": sha256(args.ffmpeg), "ffprobe_sha256": sha256(args.ffprobe), "backend": "forced CPU", "threads": 1, - "policy": "nonoverlapping30s;independent-process-per-window;auto-language;segment-times", + "harness_sha256": sha256(Path(__file__)), + "policy": "aresample16000-before-atrim-v2;samplecoverage-1ms;nonoverlapping30s;independent-process-per-window;auto-language;segment-times", "rows": rows} args.output.parent.mkdir(parents=True, exist_ok=True) args.output.write_text(json.dumps(report, indent=2, ensure_ascii=False) + "\n", encoding="utf-8") diff --git a/tools/speech-eval/test_contract.py b/tools/speech-eval/test_contract.py index 983afd6..2952550 100644 --- a/tools/speech-eval/test_contract.py +++ b/tools/speech-eval/test_contract.py @@ -4,11 +4,53 @@ from pathlib import Path import unittest import tempfile import sys -from run import segments, run_owned +import os +import wave +from run import segments, run_owned, recognized_window, decode_command, decoded_frames from score import score class ContractTests(unittest.TestCase): + @unittest.skipUnless(os.environ.get("SEARCHI_TEST_FFMPEG"), "Requires verified CPU FFmpeg") + def test_actual_ffmpeg_full_window_and_partial_tail_sample_coverage(self): + ffmpeg = Path(os.environ["SEARCHI_TEST_FFMPEG"]) + with tempfile.TemporaryDirectory() as parent: + scratch = Path(parent) + for rate in [44100, 48000]: + with self.subTest(rate=rate): + source = scratch / f"source-{rate}.wav" + with wave.open(str(source), "wb") as audio: + audio.setnchannels(1) + audio.setsampwidth(2) + audio.setframerate(rate) + audio.writeframes(b"\x00\x00" * (rate * 31)) + for offset, length, expected in [(0, 30000, 480000), (30000, 1000, 16000)]: + output = scratch / "decoded.wav" + run_owned(decode_command(ffmpeg, source, offset, length, output), scratch, + "samplecoverage", [(output, 970000)], 30) + self.assertEqual(decoded_frames(output, length), expected) + + def test_threefold_truncation_rejected(self): + with tempfile.TemporaryDirectory() as parent: + path = Path(parent) / "truncated.wav" + with wave.open(str(path), "wb") as audio: + audio.setnchannels(1) + audio.setsampwidth(2) + audio.setframerate(16000) + audio.writeframes(b"\x00\x00" * 160000) + with self.assertRaises(ValueError): + decoded_frames(path, 30000) + + def test_invalid_actual_json_is_retained_and_next_window_parses(self): + invalid = {"transcription": [{"offsets": {"from": 0, "to": 30010}, "text": "synthetic test"}]} + valid = {"transcription": [{"offsets": {"from": 0, "to": 10}, "text": "synthetic test"}]} + rows = [recognized_window(raw, offset, 30000, 1, 2) for raw, offset in [(invalid, 0), (valid, 30000)]] + self.assertEqual(rows[0]["actual_whisper_json"], invalid) + self.assertEqual(rows[0]["segments"], []) + self.assertIn("validation_error", rows[0]) + self.assertNotIn("validation_error", rows[1]) + self.assertEqual(rows[1]["segments"][0]["start_ms"], 30000) + def test_timeout_and_output_overflow_terminate_owned_child(self): for name, code, error, timeout in [ ("timeout", "import time; time.sleep(10)", TimeoutError, 0.1),