223 lines
13 KiB
Markdown
223 lines
13 KiB
Markdown
# Speech gating and timestamp research
|
||
|
||
Research recorded 2026-10-13 against whisper.cpp **v1.9.5**, exact commit
|
||
`d1be6fde11ac6e0407606b4e42fe72d34add8037`. A real-model CPU silence contract
|
||
check and complete five-source VAD run have now passed; no gated ASR comparison,
|
||
alignment experiment, product
|
||
integration or default change has been performed. The
|
||
corrected ungated ASR observations remain in [speech alignment](SPEECH-ALIGNMENT.md).
|
||
The instrumental piano's large CPU cost and unrelated text motivate testing;
|
||
they do not establish that any particular VAD model fixes the problem.
|
||
|
||
## Frozen baseline parameters
|
||
|
||
Use the pinned implementation's defaults first, before looking at new results.
|
||
Record these values explicitly in an experiment manifest rather than trusting
|
||
future upstream defaults:
|
||
|
||
| VAD option | Pinned default |
|
||
| --- | ---: |
|
||
| Enabled | false |
|
||
| Threshold | 0.5 |
|
||
| Minimum speech | 250 ms |
|
||
| Minimum silence for splitting | 100 ms |
|
||
| Maximum speech | `FLT_MAX` (no practical limit) |
|
||
| Speech padding | 30 ms |
|
||
| Samples overlap | 0.1 seconds |
|
||
|
||
Pinned CLI options include `--vad`, `--vad-model`, `--vad-threshold`,
|
||
`--vad-min-speech-duration-ms`, `--vad-min-silence-duration-ms`,
|
||
`--vad-max-speech-duration-s`, `--vad-speech-pad-ms`, and
|
||
`--vad-samples-overlap`. Detection is a speech probability decision, not a
|
||
transcription confidence threshold. [Pinned CLI source](https://github.com/ggml-org/whisper.cpp/blob/d1be6fde11ac6e0407606b4e42fe72d34add8037/examples/cli/cli.cpp#L107)
|
||
|
||
Relevant decoding defaults are temperature 0, temperature increment 0.2,
|
||
entropy threshold 2.4, log-probability threshold -1, no-speech threshold 0.6,
|
||
word timestamp threshold 0.01, unlimited segment character length, and flash
|
||
attention enabled. The greedy/beam default uses five candidates. Preserve the
|
||
ungated experiment's explicit language `auto`, one processor and CPU policy.
|
||
Changing decoding parameters together with VAD would confound the comparison.
|
||
These are inspected source defaults, not a recommended tuned policy.
|
||
|
||
## Two implementation traps
|
||
|
||
The integrated VAD path creates `whisper_vad_context_params` using the library
|
||
default: **four threads, GPU disabled**. It does not copy the transcription
|
||
`--threads` value into `n_threads`. Therefore `whisper-cli --threads 1 --vad`
|
||
does **not** prove a one-thread VAD budget. Source evidence is
|
||
`src/whisper.cpp:4588` and `:6826`. The standalone
|
||
`examples/vad-speech-segments/speech.cpp:110` explicitly copies its own CLI
|
||
thread count. Prefer that VAD-only path or a small, separately hashed diagnostic
|
||
wrapper that sets `n_threads=1` and `use_gpu=false` before loading VAD.
|
||
[Pinned library implementation](https://github.com/ggml-org/whisper.cpp/blob/d1be6fde11ac6e0407606b4e42fe72d34add8037/src/whisper.cpp#L4588)
|
||
|
||
The pinned full-JSON serializer reads raw token-data `t0/t1`; the dedicated
|
||
`whisper_full_get_token_t0/t1` accessors apply the VAD-to-original timeline map.
|
||
Their implementations differ. Segment timestamps use mapped accessors, so valid
|
||
segment offsets do not validate token offsets. Inspect CLI lines 765–789 and
|
||
library lines 8249–8318. Treat integrated-VAD token JSON as unsafe for source
|
||
seeking until a regression proves correct mapping or the diagnostic serializer
|
||
uses mapped accessors while preserving merged UTF-8 tokens.
|
||
|
||
## Candidate assets, verified without inference
|
||
|
||
The upstream downloader lists both Silero v5.1.2 and v6.2.0. The pinned README
|
||
uses v6.2.0; choose it as the **first preregistered candidate**, not because of
|
||
NASA results. Both exports were downloaded into ignored
|
||
`.searchi/speech/vad-research/` and locally verified against Hugging Face LFS
|
||
metadata. Repository revision: `9ffd54a1e1ee413ddf265af9913beaf518d1639b`.
|
||
[Pinned export files](https://huggingface.co/ggml-org/whisper-vad/tree/9ffd54a1e1ee413ddf265af9913beaf518d1639b)
|
||
|
||
| Export | Bytes | SHA-256 |
|
||
| --- | ---: | --- |
|
||
| `ggml-silero-v5.1.2.bin` | 885098 | `29940d98d42b91fbd05ce489f3ecf7c72f0a42f027e4875919a28fb4c04ea2cf` |
|
||
| `ggml-silero-v6.2.0.bin` | 885098 | `2aa269b785eeb53a82983a20501ddf7c1d9c48e33ab63a41391ac6c9f7fb6987` |
|
||
|
||
Download via `https://huggingface.co/ggml-org/whisper-vad/resolve/`
|
||
followed by the exact revision and filename, never floating `main`.
|
||
The export model card declares MIT. Silero's upstream MIT license was preserved
|
||
from commit `1e261b036686cd0017d500ee96acd1c4ba572a9d`: 1075 bytes,
|
||
SHA-256 `2e63e9a38b6e8fc0c7bc37ce174caca1862870856c6daf5697cfb785e925520b`.
|
||
The card does not provide a reproducible original-weight-to-export commit chain;
|
||
do not claim one. Distribution needs the actual license notice and a reviewed
|
||
manifest, beyond this research cache.
|
||
[Export model card](https://huggingface.co/ggml-org/whisper-vad),
|
||
[upstream license](https://github.com/snakers4/silero-vad/blob/1e261b036686cd0017d500ee96acd1c4ba572a9d/LICENSE)
|
||
|
||
## Word and token timestamps
|
||
|
||
`--output-json-full` enables token timestamps and emits token text, probability,
|
||
offsets and `t_dtw`. Tokens are subword units, potentially merged for valid UTF-8;
|
||
they are not guaranteed to be words. `--max-len` and `--split-on-word` change
|
||
segment boundaries but do not prove alignment accuracy. The upstream describes
|
||
word timestamps as experimental.
|
||
[Pinned timestamp documentation](https://github.com/ggml-org/whisper.cpp/blob/d1be6fde11ac6e0407606b4e42fe72d34add8037/README.md#word-level-timestamp-experimental)
|
||
|
||
`--dtw base` and `--dtw small` select model-specific alignment presets. They
|
||
require `--no-flash-attn`: otherwise this pinned library warns and disables DTW.
|
||
The default DTW memory budget is 128 MiB. Neither token time nor DTW repairs an
|
||
omitted or hallucinated word. Measure ordinary full JSON first, then DTW as a
|
||
separate preregistered experiment; retain process flags and actual enabled-state
|
||
logs. Do not call either forced alignment or a precise word-seeking guarantee.
|
||
|
||
## Proposed experiment and integration sequence
|
||
|
||
1. Build a CPU-only **diagnostic** VAD-only worker from the exact pinned source,
|
||
with one explicit thread. Test timestamp-unit conversion: VAD accessor/example
|
||
output is centiseconds, convert to milliseconds exactly once. Verify source
|
||
offsets, padding, overlapping intervals and 30-second boundary behavior.
|
||
2. Freeze the v6.2.0 asset and the default parameter table above before inference.
|
||
Run detection over every existing speech/music/sound competitor, preserving
|
||
all decisions and probabilities. Record no-speech completion separately from
|
||
error; zero detected speech means no ASR process, not a successful transcript.
|
||
3. Feed detected **original contiguous intervals** to unchanged ASR, retaining
|
||
their absolute source offset. Avoid concatenating disjoint speech intervals
|
||
for the first experiment; that needs a piecewise timestamp map. Deduplicate
|
||
padded overlap by source identity/times, never relevance labels. No manually
|
||
filtered piano phrases, repetitive numbers or other convenient exclusions.
|
||
4. Compare complete ungated and gated pools using the same eight unchanged NASA
|
||
queries and genuine Google text retrieval. Report VAD/decode/model-load/ASR/
|
||
tokenization/Google/ANN/publication times separately. Any processing failure
|
||
invalidates the comparison rather than shrinking the competitor pool.
|
||
5. Freeze an independent media corpus before selecting any production policy:
|
||
quiet/distant speech, accents, languages, very short utterances, speech over
|
||
instrumental music, singing, silence, synthetic effects and longer recordings.
|
||
Use independent listening/transcripts and publisher captions where possible.
|
||
Measure missed speech, false speech, hallucinated text, recognition accuracy,
|
||
timestamp error, retrieval, CPU cost and memory. Tune only on calibration;
|
||
preserve unseen held-out sources and queries.
|
||
|
||
VAD may classify instruments or singing as speech and may reject quiet words or
|
||
short phrases under the 250-ms default. That can reduce relevant retrieval recall
|
||
even while removing bad candidates. A music speech gate must never disable
|
||
Google's original audio embeddings: music/sound/image/video retrieval stays
|
||
available independently of transcription. Expected efficiency is a hypothesis;
|
||
no local speedup or false-positive rate has been measured.
|
||
|
||
If later evidence warrants production ASR, make it a resumable, bounded,
|
||
low-priority enrichment stage. Publish verified filename/text/Google media search
|
||
before transcription; retain prepared readers/query caches during enrichment.
|
||
Reuse the shared inference admission budget and yield between bounded source
|
||
intervals. Track parser/source SHA and revision through decoding, recognition
|
||
and publication; reject stale output. Persist provenance and explicit completion
|
||
or no-speech status. Foreground search must remain usable while ASR is pending;
|
||
an already executing encode cannot be claimed preemptible on a one-thread budget.
|
||
|
||
## Local diagnostic build and protocol audit
|
||
|
||
The standalone `whisper-vad-speech-segments` Release target built successfully
|
||
on Windows on 2026-10-13 from the pinned commit above, with one build job.
|
||
Its executable SHA-256 is
|
||
`2b18297bad2ea6c87fa9f314364069b74480a6c916fa8da06fe4c8ad40c9abf6`.
|
||
The inspected CMake cache has `BUILD_SHARED_LIBS`, `GGML_BACKEND_DL`,
|
||
`GGML_CUDA`, `GGML_VULKAN`, and `GGML_METAL` all OFF. This is an internal
|
||
CPU diagnostic, not a replacement for the product's hardware-detecting runtime.
|
||
Upstream C4244 and C4267 conversion warnings remain. No detection was run.
|
||
|
||
Source review identified further requirements before accepting measurements:
|
||
|
||
- Stdout interval values are rounded integer centiseconds printed with float
|
||
formatting. Multiply by ten for milliseconds; stderr instead shows seconds.
|
||
- The last inference frame is zero-padded. Its interval may exceed real audio
|
||
length. Preserve the raw interval and explicitly record its intersection with
|
||
verified source samples; do not silently clamp the evidence.
|
||
- The 30-ms speech padding is already applied. Do not apply it twice. The
|
||
segmentation implementation also merges gaps below 200 ms; the overlap
|
||
parameter does not add overlap to standalone intervals.
|
||
- A graph computation failure can break detection while the API still returns
|
||
true. Probability storage is preallocated, so count and finite values alone
|
||
cannot establish completion. Capture error-level native logs and invalidate
|
||
the run on errors before using any intervals.
|
||
- Unknown or incomplete standalone CLI arguments can exit successfully. A
|
||
successful exit alone is insufficient. A diagnostic wrapper should fix CPU
|
||
mode and one thread, check segment allocation, retain probabilities and
|
||
separate model loading, detection and segmentation timings.
|
||
|
||
The separately tracked [diagnostic wrapper](../tools/speech-vad/README.md) also
|
||
built successfully. Its SHA-256 is
|
||
`095a05c85c6f8499608ba8caa46b7dc3bbc0258a8585eeecec74ccee4b3e19d4`.
|
||
Three native contract tests passed in 1.451 seconds using the verified v6.2.0
|
||
model: 16,001 silent samples yielded all 32 expected probabilities and no speech
|
||
intervals; empty/misaligned/nonfinite/out-of-range PCM and unknown arguments
|
||
failed without JSON. The wrapper fixes CPU mode, one inference thread and the
|
||
parameter table, catches native error-level logs, and checks probability frame
|
||
coverage. This is a smoke test, not a speech-quality result or OS thread-count
|
||
measurement. An independent complete media comparison remains necessary.
|
||
|
||
## Complete frozen-pool VAD measurement
|
||
|
||
The tracked `tools/speech-vad/run.py` verified every source hash and primary
|
||
audio stream, decoded float PCM with full sample coverage, and ran the v6.2.0
|
||
candidate over all five existing competitors. Raw probabilities/logs and both
|
||
raw and source-intersected intervals are retained in ignored
|
||
`.searchi/speech/vad-v6-results.json`, SHA-256
|
||
`093be2005b57fe16a8f85161cc9da8966e70e2050cf19fa4122531e304dd70df`.
|
||
Current diagnostic worker SHA-256:
|
||
`28746d770ed470d13475d9474fcd37f28c38c39f27da09219bf9a15eef2adb19`.
|
||
|
||
| Source | Intervals | Detected source seconds | Whole VAD process ms |
|
||
| --- | ---: | ---: | ---: |
|
||
| clip04 (speech) | 12 | 66.04 | 2002.7 |
|
||
| music01 | 0 | 0 | 870.1 |
|
||
| music02 | 0 | 0 | 160.1 |
|
||
| sound01 | 0 | 0 | 57.7 |
|
||
| sound02 | 0 | 0 | 108.3 |
|
||
|
||
The first attempted run rejected finite float decoder overshoot above unity.
|
||
That input contract was incorrect: float decoding can exceed unity without
|
||
corruption. The wrapper now preserves finite samples without clipping and
|
||
rejects nonfinite values. The failure log was retained and no reduced pool
|
||
was scored. All three native contracts passed again with the corrected build;
|
||
four mapping regressions cover centiseconds, source origin, padded tails and
|
||
invalid intervals. Segmentation parameters and model remained frozen.
|
||
|
||
This small VAD-only result supports testing gated ASR next. It does not establish
|
||
speech recall, transcript correctness, timestamp accuracy, retrieval improvement
|
||
or a general music policy. No production gate was added and original Google
|
||
audio retrieval remains unchanged.
|
||
|
||
Open gates: actual one-thread audit, detection/recognition
|
||
comparison, ordinary and DTW timestamp regression, independent held-out quality,
|
||
background starvation/cancellation/crash recovery, packaging/license review,
|
||
native OS and physical GPU fallback. None is closed by this research document.
|