searchi/docs/SPEECH-VAD.md

Speech gating and timestamp research

Research recorded 2026-10-13 against whisper.cpp v1.9.5, exact commit d1be6fde11ac6e0407606b4e42fe72d34add8037. A real-model CPU silence contract check and complete five-source VAD run have now passed; no gated ASR comparison, alignment experiment, product integration or default change has been performed. The corrected ungated ASR observations remain in speech alignment. The instrumental piano's large CPU cost and unrelated text motivate testing; they do not establish that any particular VAD model fixes the problem.

Frozen baseline parameters

Use the pinned implementation's defaults first, before looking at new results. Record these values explicitly in an experiment manifest rather than trusting future upstream defaults:

VAD option	Pinned default
Enabled	false
Threshold	0.5
Minimum speech	250 ms
Minimum silence for splitting	100 ms
Maximum speech	FLT_MAX (no practical limit)
Speech padding	30 ms
Samples overlap	0.1 seconds

Pinned CLI options include --vad, --vad-model, --vad-threshold, --vad-min-speech-duration-ms, --vad-min-silence-duration-ms, --vad-max-speech-duration-s, --vad-speech-pad-ms, and --vad-samples-overlap. Detection is a speech probability decision, not a transcription confidence threshold. Pinned CLI source

Relevant decoding defaults are temperature 0, temperature increment 0.2, entropy threshold 2.4, log-probability threshold -1, no-speech threshold 0.6, word timestamp threshold 0.01, unlimited segment character length, and flash attention enabled. The greedy/beam default uses five candidates. Preserve the ungated experiment's explicit language auto, one processor and CPU policy. Changing decoding parameters together with VAD would confound the comparison. These are inspected source defaults, not a recommended tuned policy.

Two implementation traps

The integrated VAD path creates whisper_vad_context_params using the library default: four threads, GPU disabled. It does not copy the transcription --threads value into n_threads. Therefore whisper-cli --threads 1 --vad does not prove a one-thread VAD budget. Source evidence is src/whisper.cpp:4588 and :6826. The standalone examples/vad-speech-segments/speech.cpp:110 explicitly copies its own CLI thread count. Prefer that VAD-only path or a small, separately hashed diagnostic wrapper that sets n_threads=1 and use_gpu=false before loading VAD. Pinned library implementation

The pinned full-JSON serializer reads raw token-data t0/t1; the dedicated whisper_full_get_token_t0/t1 accessors apply the VAD-to-original timeline map. Their implementations differ. Segment timestamps use mapped accessors, so valid segment offsets do not validate token offsets. Inspect CLI lines 765–789 and library lines 8249–8318. Treat integrated-VAD token JSON as unsafe for source seeking until a regression proves correct mapping or the diagnostic serializer uses mapped accessors while preserving merged UTF-8 tokens.

Candidate assets, verified without inference

The upstream downloader lists both Silero v5.1.2 and v6.2.0. The pinned README uses v6.2.0; choose it as the first preregistered candidate, not because of NASA results. Both exports were downloaded into ignored .searchi/speech/vad-research/ and locally verified against Hugging Face LFS metadata. Repository revision: 9ffd54a1e1ee413ddf265af9913beaf518d1639b. Pinned export files

Export	Bytes	SHA-256
ggml-silero-v5.1.2.bin	885098	29940d98d42b91fbd05ce489f3ecf7c72f0a42f027e4875919a28fb4c04ea2cf
ggml-silero-v6.2.0.bin	885098	2aa269b785eeb53a82983a20501ddf7c1d9c48e33ab63a41391ac6c9f7fb6987

Download via https://huggingface.co/ggml-org/whisper-vad/resolve/ followed by the exact revision and filename, never floating main. The export model card declares MIT. Silero's upstream MIT license was preserved from commit 1e261b036686cd0017d500ee96acd1c4ba572a9d: 1075 bytes, SHA-256 2e63e9a38b6e8fc0c7bc37ce174caca1862870856c6daf5697cfb785e925520b. The card does not provide a reproducible original-weight-to-export commit chain; do not claim one. Distribution needs the actual license notice and a reviewed manifest, beyond this research cache. Export model card, upstream license

Word and token timestamps

--output-json-full enables token timestamps and emits token text, probability, offsets and t_dtw. Tokens are subword units, potentially merged for valid UTF-8; they are not guaranteed to be words. --max-len and --split-on-word change segment boundaries but do not prove alignment accuracy. The upstream describes word timestamps as experimental. Pinned timestamp documentation

--dtw base and --dtw small select model-specific alignment presets. They require --no-flash-attn: otherwise this pinned library warns and disables DTW. The default DTW memory budget is 128 MiB. Neither token time nor DTW repairs an omitted or hallucinated word. Measure ordinary full JSON first, then DTW as a separate preregistered experiment; retain process flags and actual enabled-state logs. Do not call either forced alignment or a precise word-seeking guarantee.

Proposed experiment and integration sequence

1. Build a CPU-only diagnostic VAD-only worker from the exact pinned source, with one explicit thread. Test timestamp-unit conversion: VAD accessor/example output is centiseconds, convert to milliseconds exactly once. Verify source offsets, padding, overlapping intervals and 30-second boundary behavior.
2. Freeze the v6.2.0 asset and the default parameter table above before inference. Run detection over every existing speech/music/sound competitor, preserving all decisions and probabilities. Record no-speech completion separately from error; zero detected speech means no ASR process, not a successful transcript.
3. Feed detected original contiguous intervals to unchanged ASR, retaining their absolute source offset. Avoid concatenating disjoint speech intervals for the first experiment; that needs a piecewise timestamp map. Deduplicate padded overlap by source identity/times, never relevance labels. No manually filtered piano phrases, repetitive numbers or other convenient exclusions.
4. Compare complete ungated and gated pools using the same eight unchanged NASA queries and genuine Google text retrieval. Report VAD/decode/model-load/ASR/ tokenization/Google/ANN/publication times separately. Any processing failure invalidates the comparison rather than shrinking the competitor pool.
5. Freeze an independent media corpus before selecting any production policy: quiet/distant speech, accents, languages, very short utterances, speech over instrumental music, singing, silence, synthetic effects and longer recordings. Use independent listening/transcripts and publisher captions where possible. Measure missed speech, false speech, hallucinated text, recognition accuracy, timestamp error, retrieval, CPU cost and memory. Tune only on calibration; preserve unseen held-out sources and queries.

VAD may classify instruments or singing as speech and may reject quiet words or short phrases under the 250-ms default. That can reduce relevant retrieval recall even while removing bad candidates. A music speech gate must never disable Google's original audio embeddings: music/sound/image/video retrieval stays available independently of transcription. Expected efficiency is a hypothesis; no local speedup or false-positive rate has been measured.

If later evidence warrants production ASR, make it a resumable, bounded, low-priority enrichment stage. Publish verified filename/text/Google media search before transcription; retain prepared readers/query caches during enrichment. Reuse the shared inference admission budget and yield between bounded source intervals. Track parser/source SHA and revision through decoding, recognition and publication; reject stale output. Persist provenance and explicit completion or no-speech status. Foreground search must remain usable while ASR is pending; an already executing encode cannot be claimed preemptible on a one-thread budget.

Local diagnostic build and protocol audit

The standalone whisper-vad-speech-segments Release target built successfully on Windows on 2026-10-13 from the pinned commit above, with one build job. Its executable SHA-256 is 2b18297bad2ea6c87fa9f314364069b74480a6c916fa8da06fe4c8ad40c9abf6. The inspected CMake cache has BUILD_SHARED_LIBS, GGML_BACKEND_DL, GGML_CUDA, GGML_VULKAN, and GGML_METAL all OFF. This is an internal CPU diagnostic, not a replacement for the product's hardware-detecting runtime. Upstream C4244 and C4267 conversion warnings remain. No detection was run.

Source review identified further requirements before accepting measurements:

• Stdout interval values are rounded integer centiseconds printed with float formatting. Multiply by ten for milliseconds; stderr instead shows seconds.
• The last inference frame is zero-padded. Its interval may exceed real audio length. Preserve the raw interval and explicitly record its intersection with verified source samples; do not silently clamp the evidence.
• The 30-ms speech padding is already applied. Do not apply it twice. The segmentation implementation also merges gaps below 200 ms; the overlap parameter does not add overlap to standalone intervals.
• A graph computation failure can break detection while the API still returns true. Probability storage is preallocated, so count and finite values alone cannot establish completion. Capture error-level native logs and invalidate the run on errors before using any intervals.
• Unknown or incomplete standalone CLI arguments can exit successfully. A successful exit alone is insufficient. A diagnostic wrapper should fix CPU mode and one thread, check segment allocation, retain probabilities and separate model loading, detection and segmentation timings.

The separately tracked diagnostic wrapper also built successfully. Its SHA-256 is 095a05c85c6f8499608ba8caa46b7dc3bbc0258a8585eeecec74ccee4b3e19d4. Three native contract tests passed in 1.451 seconds using the verified v6.2.0 model: 16,001 silent samples yielded all 32 expected probabilities and no speech intervals; empty/misaligned/nonfinite/out-of-range PCM and unknown arguments failed without JSON. The wrapper fixes CPU mode, one inference thread and the parameter table, catches native error-level logs, and checks probability frame coverage. This is a smoke test, not a speech-quality result or OS thread-count measurement. An independent complete media comparison remains necessary.

Complete frozen-pool VAD measurement

The tracked tools/speech-vad/run.py verified every source hash and primary audio stream, decoded float PCM with full sample coverage, and ran the v6.2.0 candidate over all five existing competitors. Raw probabilities/logs and both raw and source-intersected intervals are retained in ignored .searchi/speech/vad-v6-results.json, SHA-256 093be2005b57fe16a8f85161cc9da8966e70e2050cf19fa4122531e304dd70df. Current diagnostic worker SHA-256: 28746d770ed470d13475d9474fcd37f28c38c39f27da09219bf9a15eef2adb19.

Source	Intervals	Detected source seconds	Whole VAD process ms
clip04 (speech)	12	66.04	2002.7
music01	0	0	870.1
music02	0	0	160.1
sound01	0	0	57.7
sound02	0	0	108.3

The first attempted run rejected finite float decoder overshoot above unity. That input contract was incorrect: float decoding can exceed unity without corruption. The wrapper now preserves finite samples without clipping and rejects nonfinite values. The failure log was retained and no reduced pool was scored. All three native contracts passed again with the corrected build; four mapping regressions cover centiseconds, source origin, padded tails and invalid intervals. Segmentation parameters and model remained frozen.

The expanded native contract suite then passed six tests in 0.471 seconds: missing-model failures emit no report, oversized PCM is rejected before model loading, and finite decoder overshoot is accepted by actual CPU inference without input clipping. The synthetic DC overshoot input tests the input contract only and is not included in the media quality pool.

This small VAD-only result supports testing gated ASR next. It does not establish speech recall, transcript correctness, timestamp accuracy, retrieval improvement or a general music policy. No production gate was added and original Google audio retrieval remains unchanged.

Open gates: actual one-thread audit, detection/recognition comparison, ordinary and DTW timestamp regression, independent held-out quality, background starvation/cancellation/crash recovery, packaging/license review, native OS and physical GPU fallback. None is closed by this research document.