searchi/docs/SPEECH-VAD.md

# Speech gating and timestamp research

Research recorded 2026-10-13 against whisper.cpp **v1.9.5**, exact commit
`d1be6fde11ac6e0407606b4e42fe72d34add8037`. A real-model CPU silence contract
check and complete five-source VAD run have now passed; no gated ASR comparison,
alignment experiment, product
integration or default change has been performed. The
corrected ungated ASR observations remain in [speech alignment](SPEECH-ALIGNMENT.md).
The instrumental piano's large CPU cost and unrelated text motivate testing;
they do not establish that any particular VAD model fixes the problem.

## Frozen baseline parameters

Use the pinned implementation's defaults first, before looking at new results.
Record these values explicitly in an experiment manifest rather than trusting
future upstream defaults:

| VAD option | Pinned default |
| --- | ---: |
| Enabled | false |
| Threshold | 0.5 |
| Minimum speech | 250 ms |
| Minimum silence for splitting | 100 ms |
| Maximum speech | `FLT_MAX` (no practical limit) |
| Speech padding | 30 ms |
| Samples overlap | 0.1 seconds |

Pinned CLI options include `--vad`, `--vad-model`, `--vad-threshold`,
`--vad-min-speech-duration-ms`, `--vad-min-silence-duration-ms`,
`--vad-max-speech-duration-s`, `--vad-speech-pad-ms`, and
`--vad-samples-overlap`. Detection is a speech probability decision, not a
transcription confidence threshold. [Pinned CLI source](https://github.com/ggml-org/whisper.cpp/blob/d1be6fde11ac6e0407606b4e42fe72d34add8037/examples/cli/cli.cpp#L107)

Relevant decoding defaults are temperature 0, temperature increment 0.2,
entropy threshold 2.4, log-probability threshold -1, no-speech threshold 0.6,
word timestamp threshold 0.01, unlimited segment character length, and flash
attention enabled. The greedy/beam default uses five candidates. Preserve the
ungated experiment's explicit language `auto`, one processor and CPU policy.
Changing decoding parameters together with VAD would confound the comparison.
These are inspected source defaults, not a recommended tuned policy.

## Two implementation traps

The integrated VAD path creates `whisper_vad_context_params` using the library
default: **four threads, GPU disabled**. It does not copy the transcription
`--threads` value into `n_threads`. Therefore `whisper-cli --threads 1 --vad`
does **not** prove a one-thread VAD budget. Source evidence is
`src/whisper.cpp:4588` and `:6826`. The standalone
`examples/vad-speech-segments/speech.cpp:110` explicitly copies its own CLI
thread count. Prefer that VAD-only path or a small, separately hashed diagnostic
wrapper that sets `n_threads=1` and `use_gpu=false` before loading VAD.
[Pinned library implementation](https://github.com/ggml-org/whisper.cpp/blob/d1be6fde11ac6e0407606b4e42fe72d34add8037/src/whisper.cpp#L4588)

The pinned full-JSON serializer reads raw token-data `t0/t1`; the dedicated
`whisper_full_get_token_t0/t1` accessors apply the VAD-to-original timeline map.
Their implementations differ. Segment timestamps use mapped accessors, so valid
segment offsets do not validate token offsets. Inspect CLI lines 765–789 and
library lines 8249–8318. Treat integrated-VAD token JSON as unsafe for source
seeking until a regression proves correct mapping or the diagnostic serializer
uses mapped accessors while preserving merged UTF-8 tokens.

## Candidate assets, verified without inference

The upstream downloader lists both Silero v5.1.2 and v6.2.0. The pinned README
uses v6.2.0; choose it as the **first preregistered candidate**, not because of
NASA results. Both exports were downloaded into ignored
`.searchi/speech/vad-research/` and locally verified against Hugging Face LFS
metadata. Repository revision: `9ffd54a1e1ee413ddf265af9913beaf518d1639b`.
[Pinned export files](https://huggingface.co/ggml-org/whisper-vad/tree/9ffd54a1e1ee413ddf265af9913beaf518d1639b)

| Export | Bytes | SHA-256 |
| --- | ---: | --- |
| `ggml-silero-v5.1.2.bin` | 885098 | `29940d98d42b91fbd05ce489f3ecf7c72f0a42f027e4875919a28fb4c04ea2cf` |
| `ggml-silero-v6.2.0.bin` | 885098 | `2aa269b785eeb53a82983a20501ddf7c1d9c48e33ab63a41391ac6c9f7fb6987` |

Download via `https://huggingface.co/ggml-org/whisper-vad/resolve/`
followed by the exact revision and filename, never floating `main`.
The export model card declares MIT. Silero's upstream MIT license was preserved
from commit `1e261b036686cd0017d500ee96acd1c4ba572a9d`: 1075 bytes,
SHA-256 `2e63e9a38b6e8fc0c7bc37ce174caca1862870856c6daf5697cfb785e925520b`.
The card does not provide a reproducible original-weight-to-export commit chain;
do not claim one. Distribution needs the actual license notice and a reviewed
manifest, beyond this research cache.
[Export model card](https://huggingface.co/ggml-org/whisper-vad),
[upstream license](https://github.com/snakers4/silero-vad/blob/1e261b036686cd0017d500ee96acd1c4ba572a9d/LICENSE)

## Word and token timestamps

`--output-json-full` enables token timestamps and emits token text, probability,
offsets and `t_dtw`. Tokens are subword units, potentially merged for valid UTF-8;
they are not guaranteed to be words. `--max-len` and `--split-on-word` change
segment boundaries but do not prove alignment accuracy. The upstream describes
word timestamps as experimental.
[Pinned timestamp documentation](https://github.com/ggml-org/whisper.cpp/blob/d1be6fde11ac6e0407606b4e42fe72d34add8037/README.md#word-level-timestamp-experimental)

`--dtw base` and `--dtw small` select model-specific alignment presets. They
require `--no-flash-attn`: otherwise this pinned library warns and disables DTW.
The default DTW memory budget is 128 MiB. Neither token time nor DTW repairs an
omitted or hallucinated word. Measure ordinary full JSON first, then DTW as a
separate preregistered experiment; retain process flags and actual enabled-state
logs. Do not call either forced alignment or a precise word-seeking guarantee.

## Proposed experiment and integration sequence

1. Build a CPU-only **diagnostic** VAD-only worker from the exact pinned source,
   with one explicit thread. Test timestamp-unit conversion: VAD accessor/example
   output is centiseconds, convert to milliseconds exactly once. Verify source
   offsets, padding, overlapping intervals and 30-second boundary behavior.
2. Freeze the v6.2.0 asset and the default parameter table above before inference.
   Run detection over every existing speech/music/sound competitor, preserving
   all decisions and probabilities. Record no-speech completion separately from
   error; zero detected speech means no ASR process, not a successful transcript.
3. Feed detected **original contiguous intervals** to unchanged ASR, retaining
   their absolute source offset. Avoid concatenating disjoint speech intervals
   for the first experiment; that needs a piecewise timestamp map. Deduplicate
   padded overlap by source identity/times, never relevance labels. No manually
   filtered piano phrases, repetitive numbers or other convenient exclusions.
4. Compare complete ungated and gated pools using the same eight unchanged NASA
   queries and genuine Google text retrieval. Report VAD/decode/model-load/ASR/
   tokenization/Google/ANN/publication times separately. Any processing failure
   invalidates the comparison rather than shrinking the competitor pool.
5. Freeze an independent media corpus before selecting any production policy:
   quiet/distant speech, accents, languages, very short utterances, speech over
   instrumental music, singing, silence, synthetic effects and longer recordings.
   Use independent listening/transcripts and publisher captions where possible.
   Measure missed speech, false speech, hallucinated text, recognition accuracy,
   timestamp error, retrieval, CPU cost and memory. Tune only on calibration;
   preserve unseen held-out sources and queries.

VAD may classify instruments or singing as speech and may reject quiet words or
short phrases under the 250-ms default. That can reduce relevant retrieval recall
even while removing bad candidates. A music speech gate must never disable
Google's original audio embeddings: music/sound/image/video retrieval stays
available independently of transcription. Expected efficiency is a hypothesis;
no local speedup or false-positive rate has been measured.

If later evidence warrants production ASR, make it a resumable, bounded,
low-priority enrichment stage. Publish verified filename/text/Google media search
before transcription; retain prepared readers/query caches during enrichment.
Reuse the shared inference admission budget and yield between bounded source
intervals. Track parser/source SHA and revision through decoding, recognition
and publication; reject stale output. Persist provenance and explicit completion
or no-speech status. Foreground search must remain usable while ASR is pending;
an already executing encode cannot be claimed preemptible on a one-thread budget.

## Local diagnostic build and protocol audit

The standalone `whisper-vad-speech-segments` Release target built successfully
on Windows on 2026-10-13 from the pinned commit above, with one build job.
Its executable SHA-256 is
`2b18297bad2ea6c87fa9f314364069b74480a6c916fa8da06fe4c8ad40c9abf6`.
The inspected CMake cache has `BUILD_SHARED_LIBS`, `GGML_BACKEND_DL`,
`GGML_CUDA`, `GGML_VULKAN`, and `GGML_METAL` all OFF. This is an internal
CPU diagnostic, not a replacement for the product's hardware-detecting runtime.
Upstream C4244 and C4267 conversion warnings remain. No detection was run.

Source review identified further requirements before accepting measurements:

- Stdout interval values are rounded integer centiseconds printed with float
  formatting. Multiply by ten for milliseconds; stderr instead shows seconds.
- The last inference frame is zero-padded. Its interval may exceed real audio
  length. Preserve the raw interval and explicitly record its intersection with
  verified source samples; do not silently clamp the evidence.
- The 30-ms speech padding is already applied. Do not apply it twice. The
  segmentation implementation also merges gaps below 200 ms; the overlap
  parameter does not add overlap to standalone intervals.
- A graph computation failure can break detection while the API still returns
  true. Probability storage is preallocated, so count and finite values alone
  cannot establish completion. Capture error-level native logs and invalidate
  the run on errors before using any intervals.
- Unknown or incomplete standalone CLI arguments can exit successfully. A
  successful exit alone is insufficient. A diagnostic wrapper should fix CPU
  mode and one thread, check segment allocation, retain probabilities and
  separate model loading, detection and segmentation timings.

The separately tracked [diagnostic wrapper](../tools/speech-vad/README.md) also
built successfully. Its SHA-256 is
`095a05c85c6f8499608ba8caa46b7dc3bbc0258a8585eeecec74ccee4b3e19d4`.
Three native contract tests passed in 1.451 seconds using the verified v6.2.0
model: 16,001 silent samples yielded all 32 expected probabilities and no speech
intervals; empty/misaligned/nonfinite/out-of-range PCM and unknown arguments
failed without JSON. The wrapper fixes CPU mode, one inference thread and the
parameter table, catches native error-level logs, and checks probability frame
coverage. This is a smoke test, not a speech-quality result or OS thread-count
measurement. An independent complete media comparison remains necessary.

## Complete frozen-pool VAD measurement

The tracked `tools/speech-vad/run.py` verified every source hash and primary
audio stream, decoded float PCM with full sample coverage, and ran the v6.2.0
candidate over all five existing competitors. Raw probabilities/logs and both
raw and source-intersected intervals are retained in ignored
`.searchi/speech/vad-v6-results.json`, SHA-256
`093be2005b57fe16a8f85161cc9da8966e70e2050cf19fa4122531e304dd70df`.
Current diagnostic worker SHA-256:
`28746d770ed470d13475d9474fcd37f28c38c39f27da09219bf9a15eef2adb19`.

| Source | Intervals | Detected source seconds | Whole VAD process ms |
| --- | ---: | ---: | ---: |
| clip04 (speech) | 12 | 66.04 | 2002.7 |
| music01 | 0 | 0 | 870.1 |
| music02 | 0 | 0 | 160.1 |
| sound01 | 0 | 0 | 57.7 |
| sound02 | 0 | 0 | 108.3 |

The first attempted run rejected finite float decoder overshoot above unity.
That input contract was incorrect: float decoding can exceed unity without
corruption. The wrapper now preserves finite samples without clipping and
rejects nonfinite values. The failure log was retained and no reduced pool
was scored. All three native contracts passed again with the corrected build;
four mapping regressions cover centiseconds, source origin, padded tails and
invalid intervals. Segmentation parameters and model remained frozen.

The expanded native contract suite then passed six tests in 0.471 seconds:
missing-model failures emit no report, oversized PCM is rejected before model
loading, and finite decoder overshoot is accepted by actual CPU inference
without input clipping. The synthetic DC overshoot input tests the input
contract only and is not included in the media quality pool.

This small VAD-only result supports testing gated ASR next. It does not establish
speech recall, transcript correctness, timestamp accuracy, retrieval improvement
or a general music policy. No production gate was added and original Google
audio retrieval remains unchanged.

## Actual gated base transcription

The pinned base ASR worker transcribed the twelve original contiguous speech
intervals on CPU with one thread, unchanged auto-language decoding flags, and
no integrated VAD concatenation. All five competitor rows remain present;
four explicit no-speech rows launch no ASR. All timestamps and decoded coverage
passed validation. Raw ASR, logs and embedded frozen VAD evidence are retained
in `.searchi/speech/gated-base-results.json`, SHA-256
`f2af1e650af697e4540cbc3e90c0035ff560cb675186f5dad9b2afff01e53a56`.
Six harness tests and three timing-comparison tests passed.

The sequential sum of decode plus ASR process times was 250,762.6 ms in the
ungated baseline. Gated decode plus ASR **including the earlier VAD decode and
process cost** totaled 64,912.2 ms. Probe/hash/checkpoint time is not included,
these are separate runs, and this is not search latency or a general speedup.
Speech alone became slower: ASR process time rose from 23,321.7 to 58,629.4 ms
because twelve independent model starts replaced three. It yielded twenty
segments versus nineteen; segment count does not prove transcript accuracy.
Music/sound ASR text vanished through explicit no-speech decisions, not manual
phrase filtering. Original Google audio retrieval remains independent.

Genuine Google retrieval on all eight unchanged queries is now measured below;
independent speech recall/alignment assessment remains open. Do not select a production
policy from timing or this tiny corpus alone. A future persistent ASR session
could reduce model-start overhead but must be measured independently.

## Genuine Google comparison result

The updated Rust diagnostic verifies the exact embedded frozen VAD bytes,
reconstructs every source interval window and requires explicit no-speech status
for the four empty competitors. The original ungated coverage contract remains
strict. Six Rust tests passed, including omission/tampering rejection; the
agent also ran strict Clippy and built the release executable. Its recorded
SHA-256 matches the current executable:
`5a8ecb9e2ff571610cfe9f0dcd4876953d8975ff4a84972d321cd91636ceaf84`.

Actual Google EmbeddingGemma 2 CPU document/query inference completed using the
normal auto-capable worker with CPU forced. All twenty recognized segments
entered retrieval without relevance-label filtering; all five original source
identities remained verified. The complete, eligible ranking report is ignored
`.searchi/speech/gated-base-google-ranks-v2.json`, SHA-256
`1647cf5826dbeaf43f8c47209ca40af6bc2b68cd56f0c5f44eda36ed2eab8a30`.
The scorer report SHA-256 is
`3d15e72764cf2329b31caf9020af9182a91acb7229b13d7dec1b24464351c0dc`.
An initial run used a nonexistent worker path and retained its incomplete
verification checkpoint; it performed no successful inference and was not scored.

| Fixed eight-query metric | Ungated base | VAD-gated base |
| --- | ---: | ---: |
| Correct top file | 8/8 | 8/8 |
| Top start point within labelled interval | 4/8 | 4/8 |
| Top interval overlaps labelled interval | 6/8 | 6/8 |

The speech gate removed incidental music/sound transcripts and reduced this
pool's processing cost, but did **not** improve these localization metrics.
It is not selected as a production default. More independent speech coverage,
recognition and alignment work is needed before adding transcription enrichment.
The original Google media embeddings remain the product's media search path.

Open gates: actual one-thread audit, detection/recognition
comparison, ordinary and DTW timestamp regression, independent held-out quality,
background starvation/cancellation/crash recovery, packaging/license review,
native OS and physical GPU fallback. None is closed by this research document.