147 lines
8.9 KiB
Markdown
147 lines
8.9 KiB
Markdown
|
# Speech gating and timestamp research
|
|||
|
|
|
|||
|
|
Research recorded 2026-10-13 against whisper.cpp **v1.9.5**, exact commit
|
|||
|
|
`d1be6fde11ac6e0407606b4e42fe72d34add8037`. No VAD inference, alignment
|
|||
|
|
experiment, product integration or default change has been performed. The
|
|||
|
|
corrected ungated ASR observations remain in [speech alignment](SPEECH-ALIGNMENT.md).
|
|||
|
|
The instrumental piano's large CPU cost and unrelated text motivate testing;
|
|||
|
|
they do not establish that any particular VAD model fixes the problem.
|
|||
|
|
|
|||
|
|
## Frozen baseline parameters
|
|||
|
|
|
|||
|
|
Use the pinned implementation's defaults first, before looking at new results.
|
|||
|
|
Record these values explicitly in an experiment manifest rather than trusting
|
|||
|
|
future upstream defaults:
|
|||
|
|
|
|||
|
|
| VAD option | Pinned default |
|
|||
|
|
| --- | ---: |
|
|||
|
|
| Enabled | false |
|
|||
|
|
| Threshold | 0.5 |
|
|||
|
|
| Minimum speech | 250 ms |
|
|||
|
|
| Minimum silence for splitting | 100 ms |
|
|||
|
|
| Maximum speech | `FLT_MAX` (no practical limit) |
|
|||
|
|
| Speech padding | 30 ms |
|
|||
|
|
| Samples overlap | 0.1 seconds |
|
|||
|
|
|
|||
|
|
Pinned CLI options include `--vad`, `--vad-model`, `--vad-threshold`,
|
|||
|
|
`--vad-min-speech-duration-ms`, `--vad-min-silence-duration-ms`,
|
|||
|
|
`--vad-max-speech-duration-s`, `--vad-speech-pad-ms`, and
|
|||
|
|
`--vad-samples-overlap`. Detection is a speech probability decision, not a
|
|||
|
|
transcription confidence threshold. [Pinned CLI source](https://github.com/ggml-org/whisper.cpp/blob/d1be6fde11ac6e0407606b4e42fe72d34add8037/examples/cli/cli.cpp#L107)
|
|||
|
|
|
|||
|
|
Relevant decoding defaults are temperature 0, temperature increment 0.2,
|
|||
|
|
entropy threshold 2.4, log-probability threshold -1, no-speech threshold 0.6,
|
|||
|
|
word timestamp threshold 0.01, unlimited segment character length, and flash
|
|||
|
|
attention enabled. The greedy/beam default uses five candidates. Preserve the
|
|||
|
|
ungated experiment's explicit language `auto`, one processor and CPU policy.
|
|||
|
|
Changing decoding parameters together with VAD would confound the comparison.
|
|||
|
|
These are inspected source defaults, not a recommended tuned policy.
|
|||
|
|
|
|||
|
|
## Two implementation traps
|
|||
|
|
|
|||
|
|
The integrated VAD path creates `whisper_vad_context_params` using the library
|
|||
|
|
default: **four threads, GPU disabled**. It does not copy the transcription
|
|||
|
|
`--threads` value into `n_threads`. Therefore `whisper-cli --threads 1 --vad`
|
|||
|
|
does **not** prove a one-thread VAD budget. Source evidence is
|
|||
|
|
`src/whisper.cpp:4588` and `:6826`. The standalone
|
|||
|
|
`examples/vad-speech-segments/speech.cpp:110` explicitly copies its own CLI
|
|||
|
|
thread count. Prefer that VAD-only path or a small, separately hashed diagnostic
|
|||
|
|
wrapper that sets `n_threads=1` and `use_gpu=false` before loading VAD.
|
|||
|
|
[Pinned library implementation](https://github.com/ggml-org/whisper.cpp/blob/d1be6fde11ac6e0407606b4e42fe72d34add8037/src/whisper.cpp#L4588)
|
|||
|
|
|
|||
|
|
The pinned full-JSON serializer reads raw token-data `t0/t1`; the dedicated
|
|||
|
|
`whisper_full_get_token_t0/t1` accessors apply the VAD-to-original timeline map.
|
|||
|
|
Their implementations differ. Segment timestamps use mapped accessors, so valid
|
|||
|
|
segment offsets do not validate token offsets. Inspect CLI lines 765–789 and
|
|||
|
|
library lines 8249–8318. Treat integrated-VAD token JSON as unsafe for source
|
|||
|
|
seeking until a regression proves correct mapping or the diagnostic serializer
|
|||
|
|
uses mapped accessors while preserving merged UTF-8 tokens.
|
|||
|
|
|
|||
|
|
## Candidate assets, verified without inference
|
|||
|
|
|
|||
|
|
The upstream downloader lists both Silero v5.1.2 and v6.2.0. The pinned README
|
|||
|
|
uses v6.2.0; choose it as the **first preregistered candidate**, not because of
|
|||
|
|
NASA results. Both exports were downloaded into ignored
|
|||
|
|
`.searchi/speech/vad-research/` and locally verified against Hugging Face LFS
|
|||
|
|
metadata. Repository revision: `9ffd54a1e1ee413ddf265af9913beaf518d1639b`.
|
|||
|
|
[Pinned export files](https://huggingface.co/ggml-org/whisper-vad/tree/9ffd54a1e1ee413ddf265af9913beaf518d1639b)
|
|||
|
|
|
|||
|
|
| Export | Bytes | SHA-256 |
|
|||
|
|
| --- | ---: | --- |
|
|||
|
|
| `ggml-silero-v5.1.2.bin` | 885098 | `29940d98d42b91fbd05ce489f3ecf7c72f0a42f027e4875919a28fb4c04ea2cf` |
|
|||
|
|
| `ggml-silero-v6.2.0.bin` | 885098 | `2aa269b785eeb53a82983a20501ddf7c1d9c48e33ab63a41391ac6c9f7fb6987` |
|
|||
|
|
|
|||
|
|
Download via `https://huggingface.co/ggml-org/whisper-vad/resolve/`
|
|||
|
|
followed by the exact revision and filename, never floating `main`.
|
|||
|
|
The export model card declares MIT. Silero's upstream MIT license was preserved
|
|||
|
|
from commit `1e261b036686cd0017d500ee96acd1c4ba572a9d`: 1075 bytes,
|
|||
|
|
SHA-256 `2e63e9a38b6e8fc0c7bc37ce174caca1862870856c6daf5697cfb785e925520b`.
|
|||
|
|
The card does not provide a reproducible original-weight-to-export commit chain;
|
|||
|
|
do not claim one. Distribution needs the actual license notice and a reviewed
|
|||
|
|
manifest, beyond this research cache.
|
|||
|
|
[Export model card](https://huggingface.co/ggml-org/whisper-vad),
|
|||
|
|
[upstream license](https://github.com/snakers4/silero-vad/blob/1e261b036686cd0017d500ee96acd1c4ba572a9d/LICENSE)
|
|||
|
|
|
|||
|
|
## Word and token timestamps
|
|||
|
|
|
|||
|
|
`--output-json-full` enables token timestamps and emits token text, probability,
|
|||
|
|
offsets and `t_dtw`. Tokens are subword units, potentially merged for valid UTF-8;
|
|||
|
|
they are not guaranteed to be words. `--max-len` and `--split-on-word` change
|
|||
|
|
segment boundaries but do not prove alignment accuracy. The upstream describes
|
|||
|
|
word timestamps as experimental.
|
|||
|
|
[Pinned timestamp documentation](https://github.com/ggml-org/whisper.cpp/blob/d1be6fde11ac6e0407606b4e42fe72d34add8037/README.md#word-level-timestamp-experimental)
|
|||
|
|
|
|||
|
|
`--dtw base` and `--dtw small` select model-specific alignment presets. They
|
|||
|
|
require `--no-flash-attn`: otherwise this pinned library warns and disables DTW.
|
|||
|
|
The default DTW memory budget is 128 MiB. Neither token time nor DTW repairs an
|
|||
|
|
omitted or hallucinated word. Measure ordinary full JSON first, then DTW as a
|
|||
|
|
separate preregistered experiment; retain process flags and actual enabled-state
|
|||
|
|
logs. Do not call either forced alignment or a precise word-seeking guarantee.
|
|||
|
|
|
|||
|
|
## Proposed experiment and integration sequence
|
|||
|
|
|
|||
|
|
1. Build a CPU-only **diagnostic** VAD-only worker from the exact pinned source,
|
|||
|
|
with one explicit thread. Test timestamp-unit conversion: VAD accessor/example
|
|||
|
|
output is centiseconds, convert to milliseconds exactly once. Verify source
|
|||
|
|
offsets, padding, overlapping intervals and 30-second boundary behavior.
|
|||
|
|
2. Freeze the v6.2.0 asset and the default parameter table above before inference.
|
|||
|
|
Run detection over every existing speech/music/sound competitor, preserving
|
|||
|
|
all decisions and probabilities. Record no-speech completion separately from
|
|||
|
|
error; zero detected speech means no ASR process, not a successful transcript.
|
|||
|
|
3. Feed detected **original contiguous intervals** to unchanged ASR, retaining
|
|||
|
|
their absolute source offset. Avoid concatenating disjoint speech intervals
|
|||
|
|
for the first experiment; that needs a piecewise timestamp map. Deduplicate
|
|||
|
|
padded overlap by source identity/times, never relevance labels. No manually
|
|||
|
|
filtered piano phrases, repetitive numbers or other convenient exclusions.
|
|||
|
|
4. Compare complete ungated and gated pools using the same eight unchanged NASA
|
|||
|
|
queries and genuine Google text retrieval. Report VAD/decode/model-load/ASR/
|
|||
|
|
tokenization/Google/ANN/publication times separately. Any processing failure
|
|||
|
|
invalidates the comparison rather than shrinking the competitor pool.
|
|||
|
|
5. Freeze an independent media corpus before selecting any production policy:
|
|||
|
|
quiet/distant speech, accents, languages, very short utterances, speech over
|
|||
|
|
instrumental music, singing, silence, synthetic effects and longer recordings.
|
|||
|
|
Use independent listening/transcripts and publisher captions where possible.
|
|||
|
|
Measure missed speech, false speech, hallucinated text, recognition accuracy,
|
|||
|
|
timestamp error, retrieval, CPU cost and memory. Tune only on calibration;
|
|||
|
|
preserve unseen held-out sources and queries.
|
|||
|
|
|
|||
|
|
VAD may classify instruments or singing as speech and may reject quiet words or
|
|||
|
|
short phrases under the 250-ms default. That can reduce relevant retrieval recall
|
|||
|
|
even while removing bad candidates. A music speech gate must never disable
|
|||
|
|
Google's original audio embeddings: music/sound/image/video retrieval stays
|
|||
|
|
available independently of transcription. Expected efficiency is a hypothesis;
|
|||
|
|
no local speedup or false-positive rate has been measured.
|
|||
|
|
|
|||
|
|
If later evidence warrants production ASR, make it a resumable, bounded,
|
|||
|
|
low-priority enrichment stage. Publish verified filename/text/Google media search
|
|||
|
|
before transcription; retain prepared readers/query caches during enrichment.
|
|||
|
|
Reuse the shared inference admission budget and yield between bounded source
|
|||
|
|
intervals. Track parser/source SHA and revision through decoding, recognition
|
|||
|
|
and publication; reject stale output. Persist provenance and explicit completion
|
|||
|
|
or no-speech status. Foreground search must remain usable while ASR is pending;
|
|||
|
|
an already executing encode cannot be claimed preemptible on a one-thread budget.
|
|||
|
|
|
|||
|
|
Open gates: native VAD build, actual one-thread audit, detection/recognition
|
|||
|
|
comparison, ordinary and DTW timestamp regression, independent held-out quality,
|
|||
|
|
background starvation/cancellation/crash recovery, packaging/license review,
|
|||
|
|
native OS and physical GPU fallback. None is closed by this research document.
|