Freeze independent English German speech and boundary validation inputs
This commit is contained in:
karl committed
1 parent
62dd3cc5bb
commit
bc9459785d
9 files changed
+590
No files matched your search
@@ -0,0 +1,100 @@
|
||||
# Independent English/German speech preparation v1
|
||||
|
||||
No ASR, VAD or Google inference has run on this fixture. The ten topic/absence
|
||||
queries and two source recordings were frozen before preparation. The sources
|
||||
are human LibriVox readings of Article 1 of the Universal Declaration of Human
|
||||
Rights, unrelated to the earlier NASA topics. They share translated content and
|
||||
are correlated; this is not a representative multilingual speech benchmark.
|
||||
|
||||
| Source | Human reader | Recording permission | Original bytes |
|
||||
| --- | --- | --- | ---: |
|
||||
| [English Article 1](https://commons.wikimedia.org/wiki/File:Universal_Declaration_of_Human_Rights_-_eng_-_drb_-_Art1.ogg) | David Barnes | LibriVox public-domain dedication / copyrighted free use | 168186 |
|
||||
| [German Article 1](https://commons.wikimedia.org/wiki/File:Universal_Declaration_of_Human_Rights_-_ger_-_robu_-_Art1.ogg) | Roman Buettner | LibriVox public-domain dedication / copyrighted free use | 151136 |
|
||||
|
||||
The publisher pages explicitly identify the human readers, document and language.
|
||||
The primary reference text is [UN Article 1 in English](https://www.un.org/about-us/universal-declaration-of-human-rights)
|
||||
and [UNRIC Article 1 in German](https://unric.org/de/allgemeine-erklaerung-menschenrechte/).
|
||||
The manifest records reference keywords and topic paraphrases, never model
|
||||
captions or recognition-derived ground truth. Full primary text is externally
|
||||
linked; it must not be supplied as an ASR prompt or indexed as if recognized.
|
||||
|
||||
Only **319322 new bytes** were downloaded. Sources are checked against official
|
||||
Commons API size/SHA-1 and acquired SHA-256. The existing independently licensed
|
||||
US Navy drum cadence supplies the instrumental control, with its previously
|
||||
locked source bytes. Including that reused source, original inputs total
|
||||
1893869 bytes. Generated silence is exact zero PCM, released CC0.
|
||||
|
||||
## Four-clip pool and honest timestamp scope
|
||||
|
||||
Prepared audio is mono 16-kHz s16 WAV. Each human recording remains complete,
|
||||
with five seconds of exact leading silence; no reference words are cropped.
|
||||
The resulting English/German durations are 22.512 and 16.219 seconds. Music and
|
||||
silence controls each cover exactly thirty seconds. Core WAVs total 3159568 bytes.
|
||||
The music waveform is the first thirty seconds of the already locked drum source;
|
||||
no descriptive tags are used to gate recognition.
|
||||
|
||||
Actual source audio stream starts are verified as zero, and the five-second
|
||||
padding/PCM source mapping is exact. **The publishers do not provide phrase-level
|
||||
timed captions.** Neither the padding boundary nor the whole article bounds prove
|
||||
the first phoneme or each sentence's start. Strict utterance-point scoring is
|
||||
explicitly unavailable; do not count a start anywhere in the article as precise
|
||||
localization success. Independent listening/timed annotations remain required.
|
||||
|
||||
Six positive EN/DE queries describe equal dignity/rights, reason/conscience and
|
||||
brotherly treatment. Either language's recording is relevant: translated topic
|
||||
equivalence is intentional. Four absent queries ask about Mars funding and salary/
|
||||
paid leave. Silence and instrumental music remain in every candidate pool, and
|
||||
any recognized hallucination remains an explicit measured failure/competitor.
|
||||
Per-language recognized concept coverage is separate from semantic file recall.
|
||||
|
||||
## Separate seam primitives
|
||||
|
||||
`seam-plan.json` freezes two artificial repeat controls, outside the four-clip
|
||||
retrieval pool. English source samples repeat twice and German samples three
|
||||
times after the same known padding, producing 40.024 and 38.657 seconds. Their
|
||||
2517880 additional WAV bytes and exact piecewise source maps stress a 30-second
|
||||
window with two-second overlap. These joins are artificial repetitions, not
|
||||
natural uninterrupted speech. Do not extrapolate their quality to long talks.
|
||||
|
||||
Independent byte checks verified each derived piece equals the original decoded
|
||||
source PCM and the prefix is all zeros. Compare full-source VAD on these short
|
||||
controls with bounded independent windows, recording interval disagreement,
|
||||
source coverage, seam omissions/duplicates and processing cost. Restarted VAD
|
||||
need not reproduce a continuously initialized VAD's probabilities. Preserve
|
||||
differences rather than changing thresholds or treating an unverified reference
|
||||
as infallible. The same labels must not choose gate intervals or discard a seam.
|
||||
|
||||
## Preparation and acceptance
|
||||
|
||||
```powershell
|
||||
python -m unittest discover -s tools/speech-eval -p test_prepare_validation.py
|
||||
python tools/speech-eval/prepare_validation.py fixtures/speech-validation-v1/fixture.json .searchi/speech-validation-v1-seams .searchi/media-runtime/bin/ffmpeg.exe .searchi/media-runtime/bin/ffprobe.exe
|
||||
```
|
||||
|
||||
Preparation already completed; existing files are preserved, so use a fresh
|
||||
owned directory for another preparation. Five structural tests passed. Exact
|
||||
padding and every seam's source pieces were checked independently without a
|
||||
model. No recognition/quality score follows. Hashes are frozen in `inputs.lock.json`:
|
||||
|
||||
- Fixture: `e3a850d7a2385e250408f57fe51f4620ce80a20f8625ced42205f02cc46d517d`.
|
||||
- Seam plan: `5da4400814c4965fc12a87bb00c36665300730a702a69acd65a72b9f5201ac9e`.
|
||||
- Prepared `.searchi/speech-validation-v1-seams/prepared.json`:
|
||||
`085f34f0af863656a28e7a905746973a54d1123bb07053541e09c8af0a58bb3f`.
|
||||
|
||||
The earlier four-clip-only plan
|
||||
`4635e772834b0eed847459f5f4d47d7c99a591f0adba2c60ce75712557092279`
|
||||
is preserved under its original ignored directory and was never inferred.
|
||||
|
||||
Before accepted scoring, require every source/window and all ten queries, strict
|
||||
UTF-8 and bounded times, zero excluded failed competitors, unchanged worker/model/
|
||||
decoder hashes and exact source-byte provenance. Report genuine German recognition
|
||||
and all preregistered reference concepts, source-level recall/MRR, absent-return
|
||||
rate and gate false positives separately. Both negative controls must retain
|
||||
zero accepted recognized speech segments; raw Google audio search stays available.
|
||||
Measure decode/VAD/ASR/alignment/Google/storage, cold model load, foreground impact
|
||||
and warm reuse separately. ASR remains bounded to thirty seconds even for seam
|
||||
controls. Pause between work units for foreground priority; do not claim native
|
||||
preemption during one inference call.
|
||||
|
||||
This plan does not adopt a provider, change production search or close fine
|
||||
timestamp, long-source, accent, representative-noise or human-listening gates.
|
||||
@@ -0,0 +1,185 @@
|
||||
{
|
||||
"schema": 1,
|
||||
"frozen_before_inference": true,
|
||||
"assets": [
|
||||
{
|
||||
"id": "speech-en",
|
||||
"kind": "speech",
|
||||
"language": "en",
|
||||
"source_url": "https://upload.wikimedia.org/wikipedia/commons/e/e7/Universal_Declaration_of_Human_Rights_-_eng_-_drb_-_Art1.ogg",
|
||||
"source_page": "https://commons.wikimedia.org/wiki/File:Universal_Declaration_of_Human_Rights_-_eng_-_drb_-_Art1.ogg",
|
||||
"bytes": 168186,
|
||||
"sha1": "4bfeec5eab9837eff3ca8318f2824e7e6982526d",
|
||||
"license": "LibriVox public-domain dedication and copyrighted free use",
|
||||
"author": "David Barnes",
|
||||
"license_url": "https://librivox.org/pages/public-domain/",
|
||||
"reference_text_url": "https://www.un.org/about-us/universal-declaration-of-human-rights",
|
||||
"reference_section": "Article1",
|
||||
"reference_keywords": [
|
||||
"free",
|
||||
"equal",
|
||||
"dignity",
|
||||
"rights",
|
||||
"reason",
|
||||
"conscience",
|
||||
"brotherhood"
|
||||
],
|
||||
"prefix_silence_ms": 5000,
|
||||
"fine_utterance_timestamps_available": false,
|
||||
"speech_offset_note": "Exact padding5000ms does not establish first phoneme or individual sentence times."
|
||||
},
|
||||
{
|
||||
"id": "speech-de",
|
||||
"kind": "speech",
|
||||
"language": "de",
|
||||
"source_url": "https://upload.wikimedia.org/wikipedia/commons/d/d8/Universal_Declaration_of_Human_Rights_-_ger_-_robu_-_Art1.ogg",
|
||||
"source_page": "https://commons.wikimedia.org/wiki/File:Universal_Declaration_of_Human_Rights_-_ger_-_robu_-_Art1.ogg",
|
||||
"bytes": 151136,
|
||||
"sha1": "7f5d3620b4939d44849639c10b415920071353a3",
|
||||
"license": "LibriVox public-domain dedication and copyrighted free use",
|
||||
"author": "Roman Buettner",
|
||||
"license_url": "https://librivox.org/pages/public-domain/",
|
||||
"reference_text_url": "https://unric.org/de/allgemeine-erklaerung-menschenrechte/",
|
||||
"reference_section": "Article1",
|
||||
"reference_keywords": [
|
||||
"frei",
|
||||
"gleich",
|
||||
"Würde",
|
||||
"Rechten",
|
||||
"Vernunft",
|
||||
"Gewissen",
|
||||
"Brüderlichkeit"
|
||||
],
|
||||
"prefix_silence_ms": 5000,
|
||||
"fine_utterance_timestamps_available": false,
|
||||
"speech_offset_note": "Exact padding5000ms does not establish first phoneme or individual sentence times."
|
||||
}
|
||||
],
|
||||
"controls": [
|
||||
{
|
||||
"id": "music-control",
|
||||
"kind": "instrumental-negative",
|
||||
"source_fixture": "fixtures/music-validation-v1/fixture.json",
|
||||
"source_id": "unseen03",
|
||||
"source_bytes": 1574547,
|
||||
"source_sha256": "5083f07730c6ffb1cd2c3b5cc1a43714a0cfda04fe913935e83d1048466562e6",
|
||||
"license": "PD-USGov USNavy",
|
||||
"clip_duration_ms": 30000
|
||||
},
|
||||
{
|
||||
"id": "silence-control",
|
||||
"kind": "silence-negative",
|
||||
"generation": "480000 exact zero s16 samples at16kHz mono",
|
||||
"clip_duration_ms": 30000,
|
||||
"license": "CC0-1.0"
|
||||
}
|
||||
],
|
||||
"queries": [
|
||||
{
|
||||
"id": "equal-en",
|
||||
"text": "someone explains that every person has equal dignity and rights",
|
||||
"language": "en",
|
||||
"expected": [
|
||||
"speech-en",
|
||||
"speech-de"
|
||||
],
|
||||
"category": "spoken-topic",
|
||||
"strict_point_score_eligible": false
|
||||
},
|
||||
{
|
||||
"id": "equal-de",
|
||||
"text": "jemand sagt dass alle Menschen gleiche Würde und Rechte haben",
|
||||
"language": "de",
|
||||
"expected": [
|
||||
"speech-en",
|
||||
"speech-de"
|
||||
],
|
||||
"category": "spoken-topic",
|
||||
"strict_point_score_eligible": false
|
||||
},
|
||||
{
|
||||
"id": "reason-en",
|
||||
"text": "a speaker mentions reason and conscience",
|
||||
"language": "en",
|
||||
"expected": [
|
||||
"speech-en",
|
||||
"speech-de"
|
||||
],
|
||||
"category": "spoken-topic",
|
||||
"strict_point_score_eligible": false
|
||||
},
|
||||
{
|
||||
"id": "reason-de",
|
||||
"text": "eine Person spricht über Vernunft und Gewissen",
|
||||
"language": "de",
|
||||
"expected": [
|
||||
"speech-en",
|
||||
"speech-de"
|
||||
],
|
||||
"category": "spoken-topic",
|
||||
"strict_point_score_eligible": false
|
||||
},
|
||||
{
|
||||
"id": "solidarity-en",
|
||||
"text": "people should treat one another with brotherhood",
|
||||
"language": "en",
|
||||
"expected": [
|
||||
"speech-en",
|
||||
"speech-de"
|
||||
],
|
||||
"category": "spoken-topic",
|
||||
"strict_point_score_eligible": false
|
||||
},
|
||||
{
|
||||
"id": "solidarity-de",
|
||||
"text": "Menschen sollen einander mit Brüderlichkeit begegnen",
|
||||
"language": "de",
|
||||
"expected": [
|
||||
"speech-en",
|
||||
"speech-de"
|
||||
],
|
||||
"category": "spoken-topic",
|
||||
"strict_point_score_eligible": false
|
||||
},
|
||||
{
|
||||
"id": "absent-mars-en",
|
||||
"text": "a speaker explains funding for a mission to Mars",
|
||||
"language": "en",
|
||||
"expected": [],
|
||||
"category": "absent-topic",
|
||||
"strict_point_score_eligible": false
|
||||
},
|
||||
{
|
||||
"id": "absent-mars-de",
|
||||
"text": "jemand erklärt die Finanzierung einer Marsmission",
|
||||
"language": "de",
|
||||
"expected": [],
|
||||
"category": "absent-topic",
|
||||
"strict_point_score_eligible": false
|
||||
},
|
||||
{
|
||||
"id": "absent-salary-en",
|
||||
"text": "someone explains their annual salary and paid leave",
|
||||
"language": "en",
|
||||
"expected": [],
|
||||
"category": "absent-topic",
|
||||
"strict_point_score_eligible": false
|
||||
},
|
||||
{
|
||||
"id": "absent-salary-de",
|
||||
"text": "eine Person erklärt ihr Jahresgehalt und den bezahlten Urlaub",
|
||||
"language": "de",
|
||||
"expected": [],
|
||||
"category": "absent-topic",
|
||||
"strict_point_score_eligible": false
|
||||
}
|
||||
],
|
||||
"acceptance": {
|
||||
"complete_pool": "All4clips and all10queries retained; zero excluded invalid windows; invalid outputs fail accepted scoring",
|
||||
"recognition": "Report per-language transcript/reference concept coverage separately; publisher references never become inference prompts or indexed passages",
|
||||
"retrieval": "ActualGoogle textadapter, allrecognized passages inclnegativehallucinations; positive fileR1/R5+MRR and absentreturn separate",
|
||||
"timing": "No fine utterance publisher captions: precise startscore unavailable, do not credit wholeclip as precise. Independent listening/timedlabels required before fine localization acceptance",
|
||||
"gate": "Zero accepted recognized segments for exactsilence and instrumentalcontrol; retain gatingdecision/source and failures; no endpoint/rawGoogle suppression",
|
||||
"runtime": "CPU1thread on VM; ASRwindowmax30s; repeatedoperation/sourcehash guard/explicitoffsetmap; coldload separately measured"
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,119 @@
|
||||
{
|
||||
"fixture_sha256": "e3a850d7a2385e250408f57fe51f4620ce80a20f8625ced42205f02cc46d517d",
|
||||
"seam_plan_sha256": "5da4400814c4965fc12a87bb00c36665300730a702a69acd65a72b9f5201ac9e",
|
||||
"prepared_sha256": "085f34f0af863656a28e7a905746973a54d1123bb07053541e09c8af0a58bb3f",
|
||||
"frozen_before_inference": true,
|
||||
"model_inference_performed": false,
|
||||
"clips": [
|
||||
{
|
||||
"id": "speech-en",
|
||||
"sha256": "0aa1fa97d367d987bfb3bb01d7eca11db77a544552ccb8dc0a78ad74d0b496ea",
|
||||
"samples": 360192,
|
||||
"source_sha256": "d69ddbb1dbf64bae10bb03ff74edc840b08c1cd08c172ca524fa73cb2c98ee48",
|
||||
"source_audio_start_ms": 0.0,
|
||||
"prefix_silence_ms": 5000
|
||||
},
|
||||
{
|
||||
"id": "speech-de",
|
||||
"sha256": "4af09fe7d80056600e4c288161697a094108f343b069ffd581ccabc06a048afe",
|
||||
"samples": 259504,
|
||||
"source_sha256": "fb180307f9d14ced5687c6d4a59d389508d952ead568c2b83b7b8eea65e58f61",
|
||||
"source_audio_start_ms": 0.0,
|
||||
"prefix_silence_ms": 5000
|
||||
},
|
||||
{
|
||||
"id": "music-control",
|
||||
"sha256": "af76cb9e35948fcf81c72f5dcb95165c578a8bcc28d756c96aef0e3c1466b8f5",
|
||||
"samples": 480000,
|
||||
"source_audio_start_ms": 0
|
||||
},
|
||||
{
|
||||
"id": "silence-control",
|
||||
"sha256": "40abc924dc63f75aafb2400bb99fd66741e22eb9ff2450973a18db53a06286ac",
|
||||
"samples": 480000,
|
||||
"source_audio_start_ms": 0
|
||||
}
|
||||
],
|
||||
"seam_controls": [
|
||||
{
|
||||
"id": "seam-en",
|
||||
"sha256": "d4553fac71e86adbd4eced7a1c4750782c63935501446697cf5bc7b48870b31d",
|
||||
"source_sha256": "d69ddbb1dbf64bae10bb03ff74edc840b08c1cd08c172ca524fa73cb2c98ee48",
|
||||
"samples": 640384,
|
||||
"piecewise_map": [
|
||||
{
|
||||
"derived_samples": [
|
||||
80000,
|
||||
360192
|
||||
],
|
||||
"source": "speech-en",
|
||||
"source_samples": [
|
||||
0,
|
||||
280192
|
||||
],
|
||||
"copy_index": 0
|
||||
},
|
||||
{
|
||||
"derived_samples": [
|
||||
360192,
|
||||
640384
|
||||
],
|
||||
"source": "speech-en",
|
||||
"source_samples": [
|
||||
0,
|
||||
280192
|
||||
],
|
||||
"copy_index": 1
|
||||
}
|
||||
],
|
||||
"prefix_silence_samples": 80000
|
||||
},
|
||||
{
|
||||
"id": "seam-de",
|
||||
"sha256": "6b2adaab5fd505dd313b49b8878fc0b372b5141d80c589dcf06fba37ad699c00",
|
||||
"source_sha256": "fb180307f9d14ced5687c6d4a59d389508d952ead568c2b83b7b8eea65e58f61",
|
||||
"samples": 618512,
|
||||
"piecewise_map": [
|
||||
{
|
||||
"derived_samples": [
|
||||
80000,
|
||||
259504
|
||||
],
|
||||
"source": "speech-de",
|
||||
"source_samples": [
|
||||
0,
|
||||
179504
|
||||
],
|
||||
"copy_index": 0
|
||||
},
|
||||
{
|
||||
"derived_samples": [
|
||||
259504,
|
||||
439008
|
||||
],
|
||||
"source": "speech-de",
|
||||
"source_samples": [
|
||||
0,
|
||||
179504
|
||||
],
|
||||
"copy_index": 1
|
||||
},
|
||||
{
|
||||
"derived_samples": [
|
||||
439008,
|
||||
618512
|
||||
],
|
||||
"source": "speech-de",
|
||||
"source_samples": [
|
||||
0,
|
||||
179504
|
||||
],
|
||||
"copy_index": 2
|
||||
}
|
||||
],
|
||||
"prefix_silence_samples": 80000
|
||||
}
|
||||
],
|
||||
"ffmpeg_sha256": "b778ca5786ee4b0b4f37f34478617093d2fdb9a532d281fb76cda69d1d579d46",
|
||||
"ffprobe_sha256": "4e1b0afd27bf02d79739f27b2d650137470925afb19d54b5e89a708b7a389236"
|
||||
}
|
||||
@@ -0,0 +1,7 @@
|
||||
{
|
||||
"fixture_sha256": "e3a850d7a2385e250408f57fe51f4620ce80a20f8625ced42205f02cc46d517d",
|
||||
"frozen_before_inference": true,
|
||||
"source_count": 2,
|
||||
"derived_clip_count": 4,
|
||||
"query_count": 10
|
||||
}
|
||||
@@ -0,0 +1,23 @@
|
||||
{
|
||||
"schema": 1,
|
||||
"frozen_before_inference": true,
|
||||
"purpose": "Artificial repeat/offset primitive stress, separate from4clipretrievalpool; no sentence-precise labels",
|
||||
"policy": "ReferencefullsourceVAD versus independent30s VAD windows with2s overlap; no parityassumption or postscore parameter tuning",
|
||||
"recipes": [
|
||||
{
|
||||
"id": "seam-en",
|
||||
"source": "speech-en",
|
||||
"copies": 2,
|
||||
"prefix_silence_ms": 5000,
|
||||
"maximum_duration_ms": 45000
|
||||
},
|
||||
{
|
||||
"id": "seam-de",
|
||||
"source": "speech-de",
|
||||
"copies": 3,
|
||||
"prefix_silence_ms": 5000,
|
||||
"maximum_duration_ms": 45000
|
||||
}
|
||||
],
|
||||
"acceptance": "Report sourcecoverage/intervalmembership disagreement, boundary omissions/duplicates and cancellation/failure before publication; map padding exactlyonce, preserve known source offsets and no label-selected speech interval. Any whole-file probability difference is a finding, not automaticallya correctnesserror; functional acceptance requires no missing sourcewindows, validbounds and independently assessed speechrecall. Human finephoneme labels unavailable; do not claim precise localizedquality."
|
||||
}
|
||||
@@ -0,0 +1,4 @@
|
||||
{
|
||||
"seam_plan_sha256": "5da4400814c4965fc12a87bb00c36665300730a702a69acd65a72b9f5201ac9e",
|
||||
"frozen_before_inference": true
|
||||
}
|
||||
Reference in new issue
Block a user