searchi/docs/SPEECH-VAD.md

Speech gating and timestamp research

Research recorded 2026-10-13 against whisper.cpp v1.9.5, exact commit d1be6fde11ac6e0407606b4e42fe72d34add8037. No VAD inference, alignment experiment, product integration or default change has been performed. The corrected ungated ASR observations remain in speech alignment. The instrumental piano's large CPU cost and unrelated text motivate testing; they do not establish that any particular VAD model fixes the problem.

Frozen baseline parameters

Use the pinned implementation's defaults first, before looking at new results. Record these values explicitly in an experiment manifest rather than trusting future upstream defaults:

VAD option	Pinned default
Enabled	false
Threshold	0.5
Minimum speech	250 ms
Minimum silence for splitting	100 ms
Maximum speech	FLT_MAX (no practical limit)
Speech padding	30 ms
Samples overlap	0.1 seconds

Pinned CLI options include --vad, --vad-model, --vad-threshold, --vad-min-speech-duration-ms, --vad-min-silence-duration-ms, --vad-max-speech-duration-s, --vad-speech-pad-ms, and --vad-samples-overlap. Detection is a speech probability decision, not a transcription confidence threshold. Pinned CLI source

Relevant decoding defaults are temperature 0, temperature increment 0.2, entropy threshold 2.4, log-probability threshold -1, no-speech threshold 0.6, word timestamp threshold 0.01, unlimited segment character length, and flash attention enabled. The greedy/beam default uses five candidates. Preserve the ungated experiment's explicit language auto, one processor and CPU policy. Changing decoding parameters together with VAD would confound the comparison. These are inspected source defaults, not a recommended tuned policy.

Two implementation traps

The integrated VAD path creates whisper_vad_context_params using the library default: four threads, GPU disabled. It does not copy the transcription --threads value into n_threads. Therefore whisper-cli --threads 1 --vad does not prove a one-thread VAD budget. Source evidence is src/whisper.cpp:4588 and :6826. The standalone examples/vad-speech-segments/speech.cpp:110 explicitly copies its own CLI thread count. Prefer that VAD-only path or a small, separately hashed diagnostic wrapper that sets n_threads=1 and use_gpu=false before loading VAD. Pinned library implementation

The pinned full-JSON serializer reads raw token-data t0/t1; the dedicated whisper_full_get_token_t0/t1 accessors apply the VAD-to-original timeline map. Their implementations differ. Segment timestamps use mapped accessors, so valid segment offsets do not validate token offsets. Inspect CLI lines 765–789 and library lines 8249–8318. Treat integrated-VAD token JSON as unsafe for source seeking until a regression proves correct mapping or the diagnostic serializer uses mapped accessors while preserving merged UTF-8 tokens.

Candidate assets, verified without inference

The upstream downloader lists both Silero v5.1.2 and v6.2.0. The pinned README uses v6.2.0; choose it as the first preregistered candidate, not because of NASA results. Both exports were downloaded into ignored .searchi/speech/vad-research/ and locally verified against Hugging Face LFS metadata. Repository revision: 9ffd54a1e1ee413ddf265af9913beaf518d1639b. Pinned export files

Export	Bytes	SHA-256
ggml-silero-v5.1.2.bin	885098	29940d98d42b91fbd05ce489f3ecf7c72f0a42f027e4875919a28fb4c04ea2cf
ggml-silero-v6.2.0.bin	885098	2aa269b785eeb53a82983a20501ddf7c1d9c48e33ab63a41391ac6c9f7fb6987

Download via https://huggingface.co/ggml-org/whisper-vad/resolve/ followed by the exact revision and filename, never floating main. The export model card declares MIT. Silero's upstream MIT license was preserved from commit 1e261b036686cd0017d500ee96acd1c4ba572a9d: 1075 bytes, SHA-256 2e63e9a38b6e8fc0c7bc37ce174caca1862870856c6daf5697cfb785e925520b. The card does not provide a reproducible original-weight-to-export commit chain; do not claim one. Distribution needs the actual license notice and a reviewed manifest, beyond this research cache. Export model card, upstream license

Word and token timestamps

--output-json-full enables token timestamps and emits token text, probability, offsets and t_dtw. Tokens are subword units, potentially merged for valid UTF-8; they are not guaranteed to be words. --max-len and --split-on-word change segment boundaries but do not prove alignment accuracy. The upstream describes word timestamps as experimental. Pinned timestamp documentation

--dtw base and --dtw small select model-specific alignment presets. They require --no-flash-attn: otherwise this pinned library warns and disables DTW. The default DTW memory budget is 128 MiB. Neither token time nor DTW repairs an omitted or hallucinated word. Measure ordinary full JSON first, then DTW as a separate preregistered experiment; retain process flags and actual enabled-state logs. Do not call either forced alignment or a precise word-seeking guarantee.

Proposed experiment and integration sequence

1. Build a CPU-only diagnostic VAD-only worker from the exact pinned source, with one explicit thread. Test timestamp-unit conversion: VAD accessor/example output is centiseconds, convert to milliseconds exactly once. Verify source offsets, padding, overlapping intervals and 30-second boundary behavior.
2. Freeze the v6.2.0 asset and the default parameter table above before inference. Run detection over every existing speech/music/sound competitor, preserving all decisions and probabilities. Record no-speech completion separately from error; zero detected speech means no ASR process, not a successful transcript.
3. Feed detected original contiguous intervals to unchanged ASR, retaining their absolute source offset. Avoid concatenating disjoint speech intervals for the first experiment; that needs a piecewise timestamp map. Deduplicate padded overlap by source identity/times, never relevance labels. No manually filtered piano phrases, repetitive numbers or other convenient exclusions.
4. Compare complete ungated and gated pools using the same eight unchanged NASA queries and genuine Google text retrieval. Report VAD/decode/model-load/ASR/ tokenization/Google/ANN/publication times separately. Any processing failure invalidates the comparison rather than shrinking the competitor pool.
5. Freeze an independent media corpus before selecting any production policy: quiet/distant speech, accents, languages, very short utterances, speech over instrumental music, singing, silence, synthetic effects and longer recordings. Use independent listening/transcripts and publisher captions where possible. Measure missed speech, false speech, hallucinated text, recognition accuracy, timestamp error, retrieval, CPU cost and memory. Tune only on calibration; preserve unseen held-out sources and queries.

VAD may classify instruments or singing as speech and may reject quiet words or short phrases under the 250-ms default. That can reduce relevant retrieval recall even while removing bad candidates. A music speech gate must never disable Google's original audio embeddings: music/sound/image/video retrieval stays available independently of transcription. Expected efficiency is a hypothesis; no local speedup or false-positive rate has been measured.

If later evidence warrants production ASR, make it a resumable, bounded, low-priority enrichment stage. Publish verified filename/text/Google media search before transcription; retain prepared readers/query caches during enrichment. Reuse the shared inference admission budget and yield between bounded source intervals. Track parser/source SHA and revision through decoding, recognition and publication; reject stale output. Persist provenance and explicit completion or no-speech status. Foreground search must remain usable while ASR is pending; an already executing encode cannot be claimed preemptible on a one-thread budget.

Open gates: native VAD build, actual one-thread audit, detection/recognition comparison, ordinary and DTW timestamp regression, independent held-out quality, background starvation/cancellation/crash recovery, packaging/license review, native OS and physical GPU fallback. None is closed by this research document.