searchi/docs/SPEECH-VAD.md

# Speech gating and timestamp research

Research recorded 2026-10-13 against whisper.cpp **v1.9.5**, exact commit
`d1be6fde11ac6e0407606b4e42fe72d34add8037`. A real-model CPU silence contract
check has now passed; no media comparison, alignment experiment, product
integration or default change has been performed. The
corrected ungated ASR observations remain in [speech alignment](SPEECH-ALIGNMENT.md).
The instrumental piano's large CPU cost and unrelated text motivate testing;
they do not establish that any particular VAD model fixes the problem.

## Frozen baseline parameters

Use the pinned implementation's defaults first, before looking at new results.
Record these values explicitly in an experiment manifest rather than trusting
future upstream defaults:

| VAD option | Pinned default |
| --- | ---: |
| Enabled | false |
| Threshold | 0.5 |
| Minimum speech | 250 ms |
| Minimum silence for splitting | 100 ms |
| Maximum speech | `FLT_MAX` (no practical limit) |
| Speech padding | 30 ms |
| Samples overlap | 0.1 seconds |

Pinned CLI options include `--vad`, `--vad-model`, `--vad-threshold`,
`--vad-min-speech-duration-ms`, `--vad-min-silence-duration-ms`,
`--vad-max-speech-duration-s`, `--vad-speech-pad-ms`, and
`--vad-samples-overlap`. Detection is a speech probability decision, not a
transcription confidence threshold. [Pinned CLI source](https://github.com/ggml-org/whisper.cpp/blob/d1be6fde11ac6e0407606b4e42fe72d34add8037/examples/cli/cli.cpp#L107)

Relevant decoding defaults are temperature 0, temperature increment 0.2,
entropy threshold 2.4, log-probability threshold -1, no-speech threshold 0.6,
word timestamp threshold 0.01, unlimited segment character length, and flash
attention enabled. The greedy/beam default uses five candidates. Preserve the
ungated experiment's explicit language `auto`, one processor and CPU policy.
Changing decoding parameters together with VAD would confound the comparison.
These are inspected source defaults, not a recommended tuned policy.

## Two implementation traps

The integrated VAD path creates `whisper_vad_context_params` using the library
default: **four threads, GPU disabled**. It does not copy the transcription
`--threads` value into `n_threads`. Therefore `whisper-cli --threads 1 --vad`
does **not** prove a one-thread VAD budget. Source evidence is
`src/whisper.cpp:4588` and `:6826`. The standalone
`examples/vad-speech-segments/speech.cpp:110` explicitly copies its own CLI
thread count. Prefer that VAD-only path or a small, separately hashed diagnostic
wrapper that sets `n_threads=1` and `use_gpu=false` before loading VAD.
[Pinned library implementation](https://github.com/ggml-org/whisper.cpp/blob/d1be6fde11ac6e0407606b4e42fe72d34add8037/src/whisper.cpp#L4588)

The pinned full-JSON serializer reads raw token-data `t0/t1`; the dedicated
`whisper_full_get_token_t0/t1` accessors apply the VAD-to-original timeline map.
Their implementations differ. Segment timestamps use mapped accessors, so valid
segment offsets do not validate token offsets. Inspect CLI lines 765–789 and
library lines 8249–8318. Treat integrated-VAD token JSON as unsafe for source
seeking until a regression proves correct mapping or the diagnostic serializer
uses mapped accessors while preserving merged UTF-8 tokens.

## Candidate assets, verified without inference

The upstream downloader lists both Silero v5.1.2 and v6.2.0. The pinned README
uses v6.2.0; choose it as the **first preregistered candidate**, not because of
NASA results. Both exports were downloaded into ignored
`.searchi/speech/vad-research/` and locally verified against Hugging Face LFS
metadata. Repository revision: `9ffd54a1e1ee413ddf265af9913beaf518d1639b`.
[Pinned export files](https://huggingface.co/ggml-org/whisper-vad/tree/9ffd54a1e1ee413ddf265af9913beaf518d1639b)

| Export | Bytes | SHA-256 |
| --- | ---: | --- |
| `ggml-silero-v5.1.2.bin` | 885098 | `29940d98d42b91fbd05ce489f3ecf7c72f0a42f027e4875919a28fb4c04ea2cf` |
| `ggml-silero-v6.2.0.bin` | 885098 | `2aa269b785eeb53a82983a20501ddf7c1d9c48e33ab63a41391ac6c9f7fb6987` |

Download via `https://huggingface.co/ggml-org/whisper-vad/resolve/`
followed by the exact revision and filename, never floating `main`.
The export model card declares MIT. Silero's upstream MIT license was preserved
from commit `1e261b036686cd0017d500ee96acd1c4ba572a9d`: 1075 bytes,
SHA-256 `2e63e9a38b6e8fc0c7bc37ce174caca1862870856c6daf5697cfb785e925520b`.
The card does not provide a reproducible original-weight-to-export commit chain;
do not claim one. Distribution needs the actual license notice and a reviewed
manifest, beyond this research cache.
[Export model card](https://huggingface.co/ggml-org/whisper-vad),
[upstream license](https://github.com/snakers4/silero-vad/blob/1e261b036686cd0017d500ee96acd1c4ba572a9d/LICENSE)

## Word and token timestamps

`--output-json-full` enables token timestamps and emits token text, probability,
offsets and `t_dtw`. Tokens are subword units, potentially merged for valid UTF-8;
they are not guaranteed to be words. `--max-len` and `--split-on-word` change
segment boundaries but do not prove alignment accuracy. The upstream describes
word timestamps as experimental.
[Pinned timestamp documentation](https://github.com/ggml-org/whisper.cpp/blob/d1be6fde11ac6e0407606b4e42fe72d34add8037/README.md#word-level-timestamp-experimental)

`--dtw base` and `--dtw small` select model-specific alignment presets. They
require `--no-flash-attn`: otherwise this pinned library warns and disables DTW.
The default DTW memory budget is 128 MiB. Neither token time nor DTW repairs an
omitted or hallucinated word. Measure ordinary full JSON first, then DTW as a
separate preregistered experiment; retain process flags and actual enabled-state
logs. Do not call either forced alignment or a precise word-seeking guarantee.

## Proposed experiment and integration sequence

1. Build a CPU-only **diagnostic** VAD-only worker from the exact pinned source,
   with one explicit thread. Test timestamp-unit conversion: VAD accessor/example
   output is centiseconds, convert to milliseconds exactly once. Verify source
   offsets, padding, overlapping intervals and 30-second boundary behavior.
2. Freeze the v6.2.0 asset and the default parameter table above before inference.
   Run detection over every existing speech/music/sound competitor, preserving
   all decisions and probabilities. Record no-speech completion separately from
   error; zero detected speech means no ASR process, not a successful transcript.
3. Feed detected **original contiguous intervals** to unchanged ASR, retaining
   their absolute source offset. Avoid concatenating disjoint speech intervals
   for the first experiment; that needs a piecewise timestamp map. Deduplicate
   padded overlap by source identity/times, never relevance labels. No manually
   filtered piano phrases, repetitive numbers or other convenient exclusions.
4. Compare complete ungated and gated pools using the same eight unchanged NASA
   queries and genuine Google text retrieval. Report VAD/decode/model-load/ASR/
   tokenization/Google/ANN/publication times separately. Any processing failure
   invalidates the comparison rather than shrinking the competitor pool.
5. Freeze an independent media corpus before selecting any production policy:
   quiet/distant speech, accents, languages, very short utterances, speech over
   instrumental music, singing, silence, synthetic effects and longer recordings.
   Use independent listening/transcripts and publisher captions where possible.
   Measure missed speech, false speech, hallucinated text, recognition accuracy,
   timestamp error, retrieval, CPU cost and memory. Tune only on calibration;
   preserve unseen held-out sources and queries.

VAD may classify instruments or singing as speech and may reject quiet words or
short phrases under the 250-ms default. That can reduce relevant retrieval recall
even while removing bad candidates. A music speech gate must never disable
Google's original audio embeddings: music/sound/image/video retrieval stays
available independently of transcription. Expected efficiency is a hypothesis;
no local speedup or false-positive rate has been measured.

If later evidence warrants production ASR, make it a resumable, bounded,
low-priority enrichment stage. Publish verified filename/text/Google media search
before transcription; retain prepared readers/query caches during enrichment.
Reuse the shared inference admission budget and yield between bounded source
intervals. Track parser/source SHA and revision through decoding, recognition
and publication; reject stale output. Persist provenance and explicit completion
or no-speech status. Foreground search must remain usable while ASR is pending;
an already executing encode cannot be claimed preemptible on a one-thread budget.

## Local diagnostic build and protocol audit

The standalone `whisper-vad-speech-segments` Release target built successfully
on Windows on 2026-10-13 from the pinned commit above, with one build job.
Its executable SHA-256 is
`2b18297bad2ea6c87fa9f314364069b74480a6c916fa8da06fe4c8ad40c9abf6`.
The inspected CMake cache has `BUILD_SHARED_LIBS`, `GGML_BACKEND_DL`,
`GGML_CUDA`, `GGML_VULKAN`, and `GGML_METAL` all OFF. This is an internal
CPU diagnostic, not a replacement for the product's hardware-detecting runtime.
Upstream C4244 and C4267 conversion warnings remain. No detection was run.

Source review identified further requirements before accepting measurements:

- Stdout interval values are rounded integer centiseconds printed with float
  formatting. Multiply by ten for milliseconds; stderr instead shows seconds.
- The last inference frame is zero-padded. Its interval may exceed real audio
  length. Preserve the raw interval and explicitly record its intersection with
  verified source samples; do not silently clamp the evidence.
- The 30-ms speech padding is already applied. Do not apply it twice. The
  segmentation implementation also merges gaps below 200 ms; the overlap
  parameter does not add overlap to standalone intervals.
- A graph computation failure can break detection while the API still returns
  true. Probability storage is preallocated, so count and finite values alone
  cannot establish completion. Capture error-level native logs and invalidate
  the run on errors before using any intervals.
- Unknown or incomplete standalone CLI arguments can exit successfully. A
  successful exit alone is insufficient. A diagnostic wrapper should fix CPU
  mode and one thread, check segment allocation, retain probabilities and
  separate model loading, detection and segmentation timings.

The separately tracked [diagnostic wrapper](../tools/speech-vad/README.md) also
built successfully. Its SHA-256 is
`095a05c85c6f8499608ba8caa46b7dc3bbc0258a8585eeecec74ccee4b3e19d4`.
Three native contract tests passed in 1.451 seconds using the verified v6.2.0
model: 16,001 silent samples yielded all 32 expected probabilities and no speech
intervals; empty/misaligned/nonfinite/out-of-range PCM and unknown arguments
failed without JSON. The wrapper fixes CPU mode, one inference thread and the
parameter table, catches native error-level logs, and checks probability frame
coverage. This is a smoke test, not a speech-quality result or OS thread-count
measurement. An independent complete media comparison remains necessary.

Open gates: actual one-thread audit, detection/recognition
comparison, ordinary and DTW timestamp regression, independent held-out quality,
background starvation/cancellation/crash recovery, packaging/license review,
native OS and physical GPU fallback. None is closed by this research document.