karl/searchi
1
0 Fork 0
Code Issues Pull Requests Packages Projects Releases 10 Wiki Activity

Freeze normal-runtime speech timing regression without timestamp selection

Browse Source
This commit is contained in:
karl committed 2026-10-13 20:46:31 +02:00
1 parent e68f4fcadc
commit 2c3c91179b
7 files changed
+819 -31
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

tools/speech-eval/native-validation/README.md
+87
View File
@@ -51,3 +51,90 @@ comparison scoring, and hallucinations on music/silence remain measured failures
No provider, ranking weight, rejection threshold or production policy is chosen
from this small experiment. The two readings share translated subject matter and
the seam controls repeat real samples artificially.
## Completed v1 CPU execution
The frozen evaluator/source snapshot (`48f11de` / service `dab7034`, run at
`2f9190c`) completed once with the verified internal CPU v3 runtime. All four
core controls, ten queries, two seam sources and eight native source windows
were retained. Every PCM window matched the independently decoded expected
source; published recognized excerpts were byte-verified through the service.
Both cases have zero speech/indexing failures and eligible complete comparisons.
- Six positive queries retrieved a relevant file first: **6/6**, recall@5 **6/6**,
MRR **1.000**. Either translated recording is relevant under the frozen labels.
- Music and exact silence produced **zero accepted speech passages**.
- All four absent-topic queries still returned files: **4/4 no-answer failures**.
- English and German each retain all seven reference **substrings**. This is not
exact recognition quality: German contains `Artikel 1s` and
`Geisterbrüderlichkeit`, rather than the corresponding publisher phrasing.
- Seam windows preserve complete source coverage, bounds and citations, but
overlap text includes duplicate/partial or incorrect recognition, including
`Alemin`, `Einstiegleins` and `dignity and dignity`. These observations are
retained, not repaired or omitted. No precise point/localization score follows.
Core indexing took **127.61 seconds**, with **61.57 seconds** reported ASR.
Seam indexing took **163.37 seconds**, with **121.23 seconds** reported ASR.
Index timers include decoding, native work, Google and storage; native substage
times are nested, not additive with the full timer. Runtime/model verification
and query setup/warmup are separate in the summary. Normal Auto-runtime compilation
with one build job could overlap these shared Windows VM measurements; they are
not isolated hardware throughput. Eight logical CPUs and 16708012 KiB visible RAM
were exposed, but inference deliberately used one CPU thread. No normal Auto/GPU
quality or performance conclusion is inferred from this internal CPU artifact.
[Recomputed summary](evaluation-v1-summary.json) records every query outcome,
runtime/model/source hashes, cases and limitations. Immutable ignored artifacts:
| Artifact | Bytes | SHA-256 |
| --- | ---: | --- |
| `preflight.json` | 2075 | `63a0a615d37766b852c34473712156e950aeeb5a472a709bff6be00b7b26cb7a` |
| `report.json` | 174868 | `79ecb75d31c0193642c81862fd5476ad47adca518d4959e436bdd472cc281798` |
| `scored.json` | 5608 | `21d9bc84410cc63b96e2b575d99cd3c09834dfd9d2db2d0f356159df12e048e9` |
These files remain under `.searchi/qa/speech-service-validation-v1`, together
with every accepted native-call/query record and owned source/database. No
fixture, ranking parameter or provider was tuned from these results. Production
integration has meaningful real evidence here, while representative German
speech, precise publisher-timed localization, seam text handling, native UI,
broader media relevance and calibrated absence remain open.
## Frozen original media timing regression mode
`--media-v1-timing` reuses the same shared-service evaluator for the original
five complete sources: NASA `clip04`, piano `music01`, violin `music02`, bark
`sound01` and roaring effect `sound02`. All original source bytes and extensions
are preserved. This runs the actual live-window policy, not the older full-source
VAD decisions. It retains every raw media lane and every recognition output/error;
the 254-second piano's ten overlapping windows are not dropped as irrelevant.
The eight original publisher-timed speech queries/labels and fixture SHA remain
unchanged. The small additional timing policy pins the normal Auto-distribution
runtime, Google worker, decoders and existing `score.py` scoring-function hash.
Its SHA-256 is
`816b727f0447f2412b1fbe03a32ad928bdbe9c4f440bc0e7543e4b89c4dad3f0`.
Normal Auto compilation is distinct from this VM's forced CPU execution.
```powershell
cargo test --manifest-path tools/speech-eval/native-validation/Cargo.toml --locked -- --include-ignored
python -m unittest discover -s tools/speech-eval -p test_score_service_timing.py
$env:SEARCHI_FORCE_CPU = '1'
cargo run --manifest-path tools/speech-eval/native-validation/Cargo.toml --locked -- --media-v1-timing fixtures/media/fixture.json tools/speech-eval/native-validation/media-timing-policy.json C:/Users/karl/.searchi/models/embeddinggemma2 .searchi/runtime/auto/Release/searchi-gemma-worker.exe .searchi/s-a1/runtime/searchi-speech-worker.exe .searchi/speech/models/ggml-base.bin .searchi/speech/models/ggml-silero-v6.2.0.bin .searchi/media-runtime/bin/ffmpeg.exe .searchi/media-runtime/bin/ffprobe.exe .searchi/qa/speech-live-timing-v1
python tools/speech-eval/score_service_timing.py .searchi/qa/speech-live-timing-v1/report.json fixtures/media/fixture.json tools/speech-eval/native-validation/media-timing-policy.json .searchi/qa/speech-live-timing-v1/scored.json
```
The real-source preflight test verifies all five hashes and original extensions
without loading any model. Freeze/commit the evaluator policy before inference
and obtain the exclusive CPU slot. Original video/frame and music encoding can
take several minutes; this narrow timing regression does not change indexing
budgets, model/ranking policy or source files to make the run cheaper.
The adapter calls the existing publisher timing scorer's unchanged `score`
function. It uses only the actual hybrid **first result** and its byte-verified
recognized-speech excerpt's start/end. A raw audio/video first result remains an
explicit recognized-speech timing failure, with its original result retained;
no lower-ranked passage or label-selected timestamp replaces it. Failed or
missing source windows prohibit numeric accepted timing scores. Historical
full-source gated DTW **6/8** is context, never an inherited result or an isolated
causal comparison with this end-to-end hybrid run. No new timing score exists
until this exact mode executes and all checks finish.
tools/speech-eval/native-validation/evaluation-v1-summary.json
+407
View File
@@ -0,0 +1,407 @@
{
"source_commit": "2f9190c",
"evaluator_commit": "48f11de",
"service_source_commit": "dab7034",
"fixture_sha256": "e3a850d7a2385e250408f57fe51f4620ce80a20f8625ced42205f02cc46d517d",
"prepared_sha256": "085f34f0af863656a28e7a905746973a54d1123bb07053541e09c8af0a58bb3f",
"binary_sha256": "7aa75b7765b7567a3f6761c6b72449092d5adab7c3a59aaa8e55ed622340e910",
"speech_worker_sha256": "299c1b784eac46b31d228453a399e1378f327fc9ed5e34ab950c804da33e433a",
"speech_inventory_sha256": "4904995685a5e55e0a790d9cc409dc8458e39c30cda4097c5d70bf75cb5c976c",
"google_worker_sha256": "c01cb3ec13cfca722ee768e8350e1399a8b9a4d14a24a35adc3473a5cfc82a73",
"base_sha256": "60ed5bc3dd14eea856493d334349b405782ddcaf0028d4b5df4088345fba2efe",
"vad_sha256": "2aa269b785eeb53a82983a20501ddf7c1d9c48e33ab63a41391ac6c9f7fb6987",
"speech_policy": "live-window-vad-dtw-v1;silero-v6.2.0;base;30s;vad-reset;threshold0.5;minspeech250ms;minsilence100ms;maxspeechFLT_MAX;pad30ms;vad-overlap0.1s;source-sample-intersections;beam5;bestof5;temp0;tempinc0.2;entropy2.4;logprob-1;nospeech0.6;suppressblank;no-suppress-nst;auto-language;no-context;dtw-base;no-flash;contiguous-intervals;first-last-token-points;exact-utf8",
"artifact_hashes": {
"preflight.json": {
"bytes": 2075,
"sha256": "63a0a615d37766b852c34473712156e950aeeb5a472a709bff6be00b7b26cb7a"
},
"report.json": {
"bytes": 174868,
"sha256": "79ecb75d31c0193642c81862fd5476ad47adca518d4959e436bdd472cc281798"
},
"scored.json": {
"bytes": 5608,
"sha256": "21d9bc84410cc63b96e2b575d99cd3c09834dfd9d2db2d0f356159df12e048e9"
}
},
"results": {
"comparison_eligible": true,
"strict_point_score_eligible": false,
"positive_queries": 6,
"top1": 6,
"recall_at_5": 6,
"mrr": 1.0,
"negative_queries": 4,
"negative_returned_any": 4,
"native_passage_counts": {
"music-control": 0,
"silence-control": 0,
"speech-de": 3,
"speech-en": 4
},
"negative_gate_acceptance": true,
"reference_concept_substring_coverage": {
"speech-en": {
"free": true,
"equal": true,
"dignity": true,
"rights": true,
"reason": true,
"conscience": true,
"brotherhood": true
},
"speech-de": {
"frei": true,
"gleich": true,
"Würde": true,
"Rechten": true,
"Vernunft": true,
"Gewissen": true,
"Brüderlichkeit": true
}
},
"rows": [
{
"id": "equal-en",
"language": "en",
"positive": true,
"first_relevant_rank": 1,
"returned_count": 4,
"first_hit": {
"asset": "speech-de",
"match_kind": "semantic_speech",
"snippet": " Alle Menschen sind frei und gleich an Würde und Rechten geboren.",
"source": {
"kind": "timestamp_ms",
"number": 7430
}
}
},
{
"id": "equal-de",
"language": "de",
"positive": true,
"first_relevant_rank": 1,
"returned_count": 2,
"first_hit": {
"asset": "speech-de",
"match_kind": "semantic_speech",
"snippet": " Alle Menschen sind frei und gleich an Würde und Rechten geboren.",
"source": {
"kind": "timestamp_ms",
"number": 7430
}
}
},
{
"id": "reason-en",
"language": "en",
"positive": true,
"first_relevant_rank": 1,
"returned_count": 2,
"first_hit": {
"asset": "speech-en",
"match_kind": "semantic_speech",
"snippet": " they are endowed with reason and conscience.",
"source": {
"kind": "timestamp_ms",
"number": 14730
}
}
},
{
"id": "reason-de",
"language": "de",
"positive": true,
"first_relevant_rank": 1,
"returned_count": 4,
"first_hit": {
"asset": "speech-de",
"match_kind": "semantic_speech",
"snippet": " Sie sind mit Vernunft und Gewissen begabt und sollen einander im Geisterbrüderlichkeit begegnen.",
"source": {
"kind": "timestamp_ms",
"number": 11100
}
}
},
{
"id": "solidarity-en",
"language": "en",
"positive": true,
"first_relevant_rank": 1,
"returned_count": 4,
"first_hit": {
"asset": "speech-en",
"match_kind": "semantic_speech",
"snippet": " and should act towards one another in a spirit of brotherhood.",
"source": {
"kind": "timestamp_ms",
"number": 18430
}
}
},
{
"id": "solidarity-de",
"language": "de",
"positive": true,
"first_relevant_rank": 1,
"returned_count": 4,
"first_hit": {
"asset": "speech-de",
"match_kind": "semantic_speech",
"snippet": " Sie sind mit Vernunft und Gewissen begabt und sollen einander im Geisterbrüderlichkeit begegnen.",
"source": {
"kind": "timestamp_ms",
"number": 11100
}
}
},
{
"id": "absent-mars-en",
"language": "en",
"positive": false,
"first_relevant_rank": null,
"returned_count": 4,
"first_hit": {
"asset": "speech-en",
"match_kind": "semantic_speech",
"snippet": " they are endowed with reason and conscience.",
"source": {
"kind": "timestamp_ms",
"number": 14730
}
}
},
{
"id": "absent-mars-de",
"language": "de",
"positive": false,
"first_relevant_rank": null,
"returned_count": 4,
"first_hit": {
"asset": "speech-en",
"match_kind": "semantic_speech",
"snippet": " they are endowed with reason and conscience.",
"source": {
"kind": "timestamp_ms",
"number": 14730
}
}
},
{
"id": "absent-salary-en",
"language": "en",
"positive": false,
"first_relevant_rank": null,
"returned_count": 4,
"first_hit": {
"asset": "speech-en",
"match_kind": "semantic_speech",
"snippet": " they are endowed with reason and conscience.",
"source": {
"kind": "timestamp_ms",
"number": 14730
}
}
},
{
"id": "absent-salary-de",
"language": "de",
"positive": false,
"first_relevant_rank": null,
"returned_count": 4,
"first_hit": {
"asset": "speech-de",
"match_kind": "semantic_speech",
"snippet": " Alle Menschen sind frei und gleich an Würde und Rechten geboren.",
"source": {
"kind": "timestamp_ms",
"number": 7430
}
}
}
]
},
"cases": {
"core": {
"expected_native_calls": 4,
"actual_native_calls": 4,
"index_report": {
"audio_inference_calls": 4,
"chunks": 11,
"document_inference_calls": 0,
"encoded_sources": 8,
"failures": [],
"generation": 1,
"image_inference_calls": 0,
"media_failures": 0,
"media_files": 4,
"model_fingerprint": "google/embeddinggemma-2:bfcd298762cc34d0357ece5ebdd31791a3a374d8:q8_0:llama-b11461-no-logits-v1:mean:l2:query512:doc2048:windows-overlap128:prompts-v1:byte64k-overlap512-title512-v1:google/embeddinggemma-2:bfcd298762cc34d0357ece5ebdd31791a3a374d8:q8_0:llama-b11461-no-logits-v1:mtmd:mean:l2:rgb-catmullrom48-exif-largejpeg-scaled-v2:pcm16k30s-v1:image280-frame70-audio30s-overlap2-distributed240-scene120-v1:speech-v1-422af6d2bbbaaa3e943700c10035116aefcee24f35cd909bf33646bc41e00980",
"passages": 0,
"resumed": false,
"reused_chunks": 0,
"reused_sources": 0,
"speech_chunks": 7,
"speech_failures": [],
"speech_metrics": {
"admission_us": 0,
"asr_us": 61574700,
"fallback_us": 0,
"google_embedding_calls": 7,
"load_us": 209078,
"transport_us": 212365,
"vad_us": 216972
},
"speech_requested": true,
"speech_reused_sources": 0,
"speech_windows": 4
},
"indexing_ms": 127611.1518,
"speech_load_verify_ms": 3330.588,
"google_index_load_verify_ms": 7431.4775,
"google_query_load_verify_ms": 7615.0823,
"query_warmup_ms": 128.039,
"windows": [
{
"id": "speech-en",
"windows": [
{
"asset": "speech-en",
"duration_ms": 22512,
"pcm_sha256": "56948226ac5a0ad4b70523a856dad0851929c7067957f339e9a5b30fc9b369ce",
"samples": 360192,
"start_ms": 0
}
]
},
{
"id": "speech-de",
"windows": [
{
"asset": "speech-de",
"duration_ms": 16219,
"pcm_sha256": "136a1c03298a55f0ea7869e84a4684d7bd9e9fca4477b69ba860e7e300712ca2",
"samples": 259504,
"start_ms": 0
}
]
},
{
"id": "music-control",
"windows": [
{
"asset": "music-control",
"duration_ms": 30000,
"pcm_sha256": "a1fd7233bb4f9088688cfb36e449fb68b8bd5cdc70f2d193371ed9587b905193",
"samples": 480000,
"start_ms": 0
}
]
},
{
"id": "silence-control",
"windows": [
{
"asset": "silence-control",
"duration_ms": 30000,
"pcm_sha256": "124617c1f65e92d3bc895fbd869e4bb16a30754b198f59e6e973949b9aaa1b01",
"samples": 480000,
"start_ms": 0
}
]
}
]
},
"seams": {
"expected_native_calls": 4,
"actual_native_calls": 4,
"index_report": {
"audio_inference_calls": 4,
"chunks": 24,
"document_inference_calls": 0,
"encoded_sources": 4,
"failures": [],
"generation": 1,
"image_inference_calls": 0,
"media_failures": 0,
"media_files": 2,
"model_fingerprint": "google/embeddinggemma-2:bfcd298762cc34d0357ece5ebdd31791a3a374d8:q8_0:llama-b11461-no-logits-v1:mean:l2:query512:doc2048:windows-overlap128:prompts-v1:byte64k-overlap512-title512-v1:google/embeddinggemma-2:bfcd298762cc34d0357ece5ebdd31791a3a374d8:q8_0:llama-b11461-no-logits-v1:mtmd:mean:l2:rgb-catmullrom48-exif-largejpeg-scaled-v2:pcm16k30s-v1:image280-frame70-audio30s-overlap2-distributed240-scene120-v1:speech-v1-422af6d2bbbaaa3e943700c10035116aefcee24f35cd909bf33646bc41e00980",
"passages": 0,
"resumed": false,
"reused_chunks": 0,
"reused_sources": 0,
"speech_chunks": 20,
"speech_failures": [],
"speech_metrics": {
"admission_us": 0,
"asr_us": 121231168,
"fallback_us": 0,
"google_embedding_calls": 20,
"load_us": 213591,
"transport_us": 109395,
"vad_us": 187061
},
"speech_requested": true,
"speech_reused_sources": 0,
"speech_windows": 4
},
"indexing_ms": 163367.8024,
"speech_load_verify_ms": 3011.0661,
"google_index_load_verify_ms": 7365.343099999999,
"google_query_load_verify_ms": 7039.3844,
"query_warmup_ms": 115.2243,
"windows": [
{
"id": "seam-en",
"windows": [
{
"asset": "seam-en",
"duration_ms": 30000,
"pcm_sha256": "7b9e755453e90f67f4ee9f1a6c4aaa5048225faeb266e9e187c890baac6a7138",
"samples": 480000,
"start_ms": 0
},
{
"asset": "seam-en",
"duration_ms": 12024,
"pcm_sha256": "40b1bac0855bee11643dcce5d2e38801c5d3533993bd035f6a3c70ef219f180f",
"samples": 192384,
"start_ms": 28000
}
]
},
{
"id": "seam-de",
"windows": [
{
"asset": "seam-de",
"duration_ms": 30000,
"pcm_sha256": "9b77a526b62d1f473a014626eb312f995b3dd4b788c96bc31981edd243c50eea",
"samples": 480000,
"start_ms": 0
},
{
"asset": "seam-de",
"duration_ms": 10657,
"pcm_sha256": "5d5c5988ab40601479b72498e7a63dbfedfe0567572597da52811b63c0a5d23b",
"samples": 170512,
"start_ms": 28000
}
]
}
]
}
},
"limitations": [
"InternalCPUv3 runtime; normalAuto/GPU behavior not evaluated",
"Native normalAuto compilation jobs1 overlapped sharedVM timings; no controlled hardware throughput claim",
"Translated content correlated; concept substrings are not WER or exact transcript fidelity",
"German Geisterbrüderlichkeit and Artikel1s recognized incorrectly; seam partial/duplicate text retained",
"No timed publisher phrase groundtruth: strictpoint scoring unavailable",
"All4 noanswer queries return files, no abstention improvement",
"Accepted publicSpeechTranscription values recorded, no native wire/stderr capture"
],
"production_policy_changed": false
}
tools/speech-eval/native-validation/media-timing-policy.json
+21
View File
@@ -0,0 +1,21 @@
{
"schema": 1,
"policy": "frozen-media-v1-live-speech-timing-v1",
"fixture_sha256": "b4e465a0eb9582b826eebd2a20797e1680e4907e336d02d0ef2194e2ba0af393",
"sources": ["clip04", "music01", "music02", "sound01", "sound02"],
"query_category": "video-speech",
"query_count": 8,
"scoring_function_sha256": "8049d82a3a10e85640e10a6c3dd4b3657c677293e2a34356d833f5e3a8317628",
"source_max_duration_ms": 300000,
"source_max_windows": 12,
"speech_worker_sha256": "0b716fa17e8216ed46c7755d15acab20f658254b99780c7a68bab03d765c55bc",
"speech_inventory_sha256": "0e116c604141af147bd91736ec0722a0660dcda8c9da008e494c464753f4e5a1",
"google_worker_sha256": "c01cb3ec13cfca722ee768e8350e1399a8b9a4d14a24a35adc3473a5cfc82a73",
"ffmpeg_sha256": "b778ca5786ee4b0b4f37f34478617093d2fdb9a532d281fb76cda69d1d579d46",
"ffprobe_sha256": "4e1b0afd27bf02d79739f27b2d650137470925afb19d54b5e89a708b7a389236",
"speech_runtime": "normal auto distribution compiled with Vulkan; forced CPU execution in Windows VM",
"ranking": "Shared service hybrid first result only; retain every source, native window, published speech passage and raw media lane; no label-selected passages, weights or cutoff",
"scoring": "Reuse unchanged eight publisher-timed labels through existing score.py score function; non-speech first results explicitly fail recognized-speech point/overlap, retain them in diagnostic rows; no source pool exclusions",
"prior_score": "Historical full-source gated DTW 6/8 is context only; cannot carry over to live-window or establish isolated policy causality",
"inference": "Actual SpeechWorker VAD/ASR and Google EmbeddingGemma2, CPU one thread; no simulated embeddings"
}
tools/speech-eval/native-validation/src/main.rs
+171 -31
View File
@@ -15,6 +15,35 @@ use std::{
time::Instant,
};
const TIMING_POLICY_SHA: &str = "816b727f0447f2412b1fbe03a32ad928bdbe9c4f440bc0e7543e4b89c4dad3f0";
const MEDIA_FIXTURE_SHA: &str = "b4e465a0eb9582b826eebd2a20797e1680e4907e336d02d0ef2194e2ba0af393";
fn timing_sources(fixture: &Value, root: &Path) -> Result<Vec<Value>> {
let mut values = Vec::new();
for id in ["clip04", "music01", "music02", "sound01", "sound02"] {
let matches: Vec<_> = fixture["assets"]
.as_array()
.context("Fixture assets missing")?
.iter()
.filter(|asset| asset["id"] == id)
.collect();
ensure!(
matches.len() == 1,
"Required timing source missing or duplicated"
);
let asset = matches[0];
let path = root.join(asset["path"].as_str().context("Asset path missing")?);
ensure!(
fs::metadata(&path)?.len()
== asset["bytes"].as_u64().context("Asset byte size missing")?
&& file_hash(&path)? == asset["sha256"],
"Timing source bytes changed"
);
values.push(json!({"id":id,"path":path,"sha256":asset["sha256"]}));
}
Ok(values)
}
fn hash(bytes: &[u8]) -> String {
format!("{:x}", Sha256::digest(bytes))
}
@@ -40,14 +69,14 @@ fn write_new(path: &Path, value: &Value) -> Result<()> {
file.sync_all()?;
Ok(())
}
fn windows(start: u32, end: u32) -> Result<Vec<AudioWindow>> {
fn windows(start: u32, end: u32, limit: usize) -> Result<Vec<AudioWindow>> {
ensure!(start <= end, "Invalid source range");
let mut values = Vec::new();
let mut next = start;
while next < end {
ensure!(
values.len() < 4,
"Mini fixture exceeded bounded window count"
values.len() < limit,
"Fixture exceeded bounded window count"
);
values.push(AudioWindow {
start_ms: next,
@@ -59,8 +88,8 @@ fn windows(start: u32, end: u32) -> Result<Vec<AudioWindow>> {
next = next.checked_add(28000).context("Source offset overflow")?;
}
ensure!(
values.len() <= 4,
"Mini fixture exceeded bounded window count"
values.len() <= limit,
"Fixture exceeded bounded window count"
);
Ok(values)
}
@@ -118,6 +147,8 @@ fn run_case(
queries: &[Value],
assets: &Assets,
output: &Path,
max_windows: usize,
point_labels: bool,
) -> Result<Value> {
let case = output.join(name);
fs::create_dir(&case)?;
@@ -133,11 +164,15 @@ fn run_case(
source["sha256"] == file_hash(&input)?,
"Frozen prepared source changed"
);
let path = root.join(format!("{id}.wav"));
let extension = input
.extension()
.and_then(|value| value.to_str())
.context("Original extension missing")?;
let path = root.join(format!("{id}.{extension}"));
fs::copy(&input, &path)?;
let metadata = probe(&assets.tools, &path, &limits)?;
let mut source_windows = Vec::new();
for window in windows(metadata.audio_start_ms, metadata.audio_end_ms)? {
for window in windows(metadata.audio_start_ms, metadata.audio_end_ms, max_windows)? {
let pcm = decode_audio_aligned(&assets.tools, &path, window, &limits)?;
ensure!(
pcm.len() == window.duration_ms as usize * 16,
@@ -241,7 +276,7 @@ fn run_case(
};
hits.push(json!({"asset":id,"result":hit,"verified_recognized_excerpt":excerpt}));
}
let row = json!({"query":query,"hits":hits,"metrics":response.metrics,"strict_point_score_eligible":false});
let row = json!({"query":query,"hits":hits,"metrics":response.metrics,"strict_point_score_eligible":point_labels});
write_new(
&case.join(format!(
"query-{}.json",
@@ -257,12 +292,18 @@ fn run_case(
"Source changed during evaluation"
);
}
let report = json!({"name":name,"sources":source_rows,"lexical_report":discovery,"index_report":indexed,"expected_native_calls":expected_calls,"native_calls":recorded,"published_speech":chunks,"verified_citations":citations,"rows":rows,"native_fingerprint":fingerprint,"google_index_worker":google_index_worker,"google_query_worker":reader.worker_info(),"lexical_ms":lexical_ms,"speech_load_verify_ms":speech_load_ms,"google_index_load_verify_ms":google_index_load_ms,"indexing_ms":indexing_ms,"google_query_load_verify_ms":google_query_load_ms,"query_warmup_ms":warmup_ms,"comparison_eligible":complete_calls&&indexed.speech_failures.is_empty()&&indexed.failures.is_empty(),"strict_point_score_eligible":false});
let report = json!({"name":name,"sources":source_rows,"lexical_report":discovery,"index_report":indexed,"expected_native_calls":expected_calls,"native_calls":recorded,"published_speech":chunks,"verified_citations":citations,"rows":rows,"native_fingerprint":fingerprint,"google_index_worker":google_index_worker,"google_query_worker":reader.worker_info(),"lexical_ms":lexical_ms,"speech_load_verify_ms":speech_load_ms,"google_index_load_verify_ms":google_index_load_ms,"indexing_ms":indexing_ms,"google_query_load_verify_ms":google_query_load_ms,"query_warmup_ms":warmup_ms,"comparison_eligible":complete_calls&&indexed.speech_failures.is_empty()&&indexed.failures.is_empty(),"strict_point_score_eligible":point_labels});
write_new(&case.join("report.json"), &report)?;
Ok(report)
}
fn main() -> Result<()> {
let args: Vec<_> = std::env::args_os().skip(1).collect();
let mut args: Vec<_> = std::env::args_os().skip(1).collect();
let timing = args
.first()
.is_some_and(|value| value == "--media-v1-timing");
if timing {
args.remove(0);
}
ensure!(args.len()==10,"Supply FIXTURE PREPARED GOOGLE_MODELS GOOGLE_WORKER SPEECH_WORKER BASE VAD FFMPEG FFPROBE NEW_OUTPUT_DIRECTORY");
ensure!(
std::env::var("SEARCHI_FORCE_CPU").as_deref() == Ok("1"),
@@ -271,16 +312,30 @@ fn main() -> Result<()> {
let fixture_path = PathBuf::from(&args[0]);
let fixture_raw = fs::read(&fixture_path)?;
let fixture: Value = serde_json::from_slice(&fixture_raw)?;
let input_lock: Value =
serde_json::from_slice(&fs::read(fixture_path.with_file_name("inputs.lock.json"))?)?;
let prepared_raw = fs::read(&args[1])?;
let prepared: Value = serde_json::from_slice(&prepared_raw)?;
ensure!(
input_lock["fixture_sha256"] == hash(&fixture_raw)
&& input_lock["prepared_sha256"] == hash(&prepared_raw)
&& prepared["fixture_sha256"] == hash(&fixture_raw),
"Frozen fixture/prepared plan differs"
);
let prepared: Value = if timing {
ensure!(
hash(&fixture_raw) == MEDIA_FIXTURE_SHA && hash(&prepared_raw) == TIMING_POLICY_SHA,
"Frozen media-v1 timing policy or labels changed"
);
let policy: Value = serde_json::from_slice(&prepared_raw)?;
let clips = timing_sources(
&fixture,
fixture_path.parent().context("Fixture root absent")?,
)?;
json!({"clips":clips,"policy":policy})
} else {
let input_lock: Value =
serde_json::from_slice(&fs::read(fixture_path.with_file_name("inputs.lock.json"))?)?;
let value: Value = serde_json::from_slice(&prepared_raw)?;
ensure!(
input_lock["fixture_sha256"] == hash(&fixture_raw)
&& input_lock["prepared_sha256"] == hash(&prepared_raw)
&& value["fixture_sha256"] == hash(&fixture_raw),
"Frozen fixture/prepared plan differs"
);
value
};
let assets = Assets {
model: PathBuf::from(&args[2]),
google: PathBuf::from(&args[3]),
@@ -289,11 +344,32 @@ fn main() -> Result<()> {
vad: PathBuf::from(&args[6]),
tools: FfmpegTools::new(PathBuf::from(&args[7]), PathBuf::from(&args[8]))?,
};
ensure!(
prepared["ffmpeg_sha256"] == file_hash(&assets.tools.ffmpeg)?
&& prepared["ffprobe_sha256"] == file_hash(&assets.tools.ffprobe)?,
"Frozen decoder differs"
);
if timing {
ensure!(
prepared["policy"]["speech_worker_sha256"] == file_hash(&assets.speech)?
&& prepared["policy"]["speech_inventory_sha256"]
== file_hash(
&assets
.speech
.parent()
.context("Runtime root")?
.join("runtime-manifest.json")
)?,
"Frozen normal runtime changed"
);
ensure!(
prepared["policy"]["google_worker_sha256"] == file_hash(&assets.google)?
&& prepared["policy"]["ffmpeg_sha256"] == file_hash(&assets.tools.ffmpeg)?
&& prepared["policy"]["ffprobe_sha256"] == file_hash(&assets.tools.ffprobe)?,
"Frozen Google worker or decoder changed"
);
} else {
ensure!(
prepared["ffmpeg_sha256"] == file_hash(&assets.tools.ffmpeg)?
&& prepared["ffprobe_sha256"] == file_hash(&assets.tools.ffprobe)?,
"Frozen decoder differs"
);
}
let output = PathBuf::from(&args[9]);
fs::create_dir(&output)?;
let mut provenance = BTreeMap::new();
@@ -308,15 +384,51 @@ fn main() -> Result<()> {
] {
provenance.insert(file, file_hash(Path::new(file))?);
}
let source_clock_policy = if timing {
"ActualService speech30s overlap2s; independentPCM window hashes; original frozen publisher timed labels"
} else {
"ActualService speech30s overlap2s; independently decodedPCM window hashes; no strict word labels"
};
write_new(
&output.join("preflight.json"),
&json!({"fixture_sha256":hash(&fixture_raw),"prepared_sha256":hash(&prepared_raw),"binary_sha256":file_hash(&std::env::current_exe()?)?,"speech_worker_sha256":file_hash(&assets.speech)?,"speech_inventory_sha256":file_hash(&assets.speech.parent().context("Runtime directory")?.join("runtime-manifest.json"))?,"base_sha256":file_hash(&assets.base)?,"vad_sha256":file_hash(&assets.vad)?,"google_worker_sha256":file_hash(&assets.google)?,"source_hashes_at_start":provenance,"speech_policy":searchi_speech::POLICY,"source_clock_policy":"ActualService speech30s overlap2s; independently decodedPCM window hashes; no strict word labels","force_cpu":true,"threads":1}),
&json!({"fixture_sha256":hash(&fixture_raw),"prepared_sha256":hash(&prepared_raw),"binary_sha256":file_hash(&std::env::current_exe()?)?,"speech_worker_sha256":file_hash(&assets.speech)?,"speech_inventory_sha256":file_hash(&assets.speech.parent().context("Runtime directory")?.join("runtime-manifest.json"))?,"base_sha256":file_hash(&assets.base)?,"vad_sha256":file_hash(&assets.vad)?,"google_worker_sha256":file_hash(&assets.google)?,"ffmpeg_sha256":file_hash(&assets.tools.ffmpeg)?,"ffprobe_sha256":file_hash(&assets.tools.ffprobe)?,"source_hashes_at_start":provenance,"speech_policy":searchi_speech::POLICY,"source_clock_policy":source_clock_policy,"force_cpu":true,"threads":1}),
)?;
if timing {
write_new(&output.join("timing-policy.json"), &prepared["policy"])?;
let sources = prepared["clips"]
.as_array()
.context("Five timing sources absent")?;
let queries: Vec<_> = fixture["queries"]
.as_array()
.context("Queries absent")?
.iter()
.filter(|query| query["category"] == "video-speech")
.cloned()
.collect();
ensure!(
queries.len() == 8,
"All eight frozen timing queries required"
);
let result = match run_case("timing", sources, &queries, &assets, &output, 12, true) {
Ok(value) => value,
Err(error) => {
write_new(
&output.join("failure.json"),
&json!({"complete":false,"error":format!("{error:#}"),"comparison_eligible":false}),
)?;
return Err(error);
}
};
return write_new(
&output.join("report.json"),
&json!({"complete":true,"fixture_sha256":hash(&fixture_raw),"timing_policy_sha256":TIMING_POLICY_SHA,"timing":result,"strict_point_score_eligible":true,"production_policy_changed":false,"scope":"Actual shared service hybrid first hits, same completefive-source timingpool; normalauto speech runtime forcedCPU; live30s/2soverlap; no label selected spans or carried-over score"}),
);
}
let core = prepared["clips"].as_array().context("Core clips missing")?;
ensure!(core.len() == 4, "All four controls required");
let queries = fixture["queries"].as_array().context("Queries missing")?;
ensure!(queries.len() == 10, "All ten queries required");
let first = match run_case("core", core, queries, &assets, &output) {
let first = match run_case("core", core, queries, &assets, &output, 4, false) {
Ok(value) => value,
Err(error) => {
write_new(
@@ -330,7 +442,7 @@ fn main() -> Result<()> {
.as_array()
.context("Seam controls missing")?;
ensure!(seams.len() == 2, "Both seam sources required");
let second = match run_case("seams", seams, &[], &assets, &output) {
let second = match run_case("seams", seams, &[], &assets, &output, 4, false) {
Ok(value) => value,
Err(error) => {
write_new(
@@ -350,21 +462,49 @@ mod tests {
use super::*;
#[test]
fn overlap_maps_complete_seam_tail() {
let value = windows(0, 40024).unwrap();
let value = windows(0, 40024, 4).unwrap();
assert_eq!(value.len(), 2);
assert_eq!(value[1].start_ms, 28000);
assert_eq!(value[1].duration_ms, 12024);
}
#[test]
fn positioned_short_source_keeps_origin() {
let value = windows(2000, 4000).unwrap();
let value = windows(2000, 4000, 4).unwrap();
assert_eq!(value[0].start_ms, 2000);
assert_eq!(value[0].duration_ms, 2000);
}
#[test]
fn invalid_or_large_fixture_ranges_fail() {
assert!(windows(3, 2).is_err());
assert!(windows(0, 200000).is_err());
assert!(windows(3, 2, 4).is_err());
assert!(windows(0, 200000, 4).is_err());
}
#[test]
fn complete_piano_timing_windows_remain_bounded() {
let values = windows(0, 254454, 12).unwrap();
assert_eq!(values.len(), 10);
assert_eq!(values.last().unwrap().start_ms, 252000);
assert_eq!(values.last().unwrap().duration_ms, 2454);
assert!(values.iter().all(|window| window.duration_ms <= 30000));
}
#[test]
#[ignore = "Requires the five hash-verified real media fixture files; no inference"]
fn timing_mode_preflight_checks_all_real_frozen_sources_without_models() -> Result<()> {
let root = Path::new(env!("CARGO_MANIFEST_DIR")).join("../../../fixtures/media");
let raw = fs::read(root.join("fixture.json"))?;
ensure!(hash(&raw) == MEDIA_FIXTURE_SHA, "Frozen fixture changed");
let fixture: Value = serde_json::from_slice(&raw)?;
let values = timing_sources(&fixture, &root)?;
assert_eq!(values.len(), 5);
assert!(values
.iter()
.any(|source| Path::new(source["path"].as_str().unwrap())
.extension()
.is_some_and(|value| value == "mp4")));
ensure!(
hash(include_bytes!("../media-timing-policy.json")) == TIMING_POLICY_SHA,
"Frozen timing policy changed"
);
Ok(())
}
#[test]
fn existing_raw_output_is_preserved() -> Result<()> {
tools/speech-eval/score_service_timing.py
+79
View File
@@ -0,0 +1,79 @@
"""Adapter to unchanged publisher timing scorer; actual service top hit, no oracle."""
import argparse
from collections import Counter
import hashlib
import json
from pathlib import Path
from score import score as publisher_score
from run import FIXTURE_SHA
SOURCES = {"clip04", "music01", "music02", "sound01", "sound02"}
def evaluate(report, fixture):
if report.get("complete") is not True or report.get("fixture_sha256") != FIXTURE_SHA:
raise ValueError("Incomplete/stale timing report")
case = report["timing"]
if {source["id"] for source in case["sources"]} != SOURCES or len(case["sources"]) != 5:
raise ValueError("All five competitors are mandatory")
expected_calls = Counter(window["pcm_sha256"] for source in case["sources"] for window in source["windows"])
actual_calls = Counter(call["pcm_sha256"] for call in case["native_calls"])
eligible = (case["comparison_eligible"] is True and expected_calls == actual_calls
and all("error" not in call for call in case["native_calls"]))
frozen = {query["id"]: query for query in fixture["queries"] if query["category"] == "video-speech"}
if len(frozen) != 8 or len(case["rows"]) != 8 or {row["query"]["id"] for row in case["rows"]} != set(frozen):
raise ValueError("All eight frozen queries required")
ranked = {}
diagnostics = []
for row in case["rows"]:
query = row["query"]
if query != frozen[query["id"]]:
raise ValueError("Frozen timing labels/text changed")
hits = row["hits"]
if not {hit["asset"] for hit in hits} <= SOURCES:
raise ValueError("Unexpected competitor")
first = hits[0] if hits else None
diagnostic = {"query": query["id"], "first_hit": first, "first_result_kind": None if not first else first["result"]["match_kind"], "returned_count": len(hits), "recognized_speech_first": False}
ranked[query["id"]] = []
if first and first["result"]["match_kind"] == "semantic_speech":
excerpt = first.get("verified_recognized_excerpt")
source = first["result"].get("source")
if not excerpt or not source or source["kind"] != "timestamp_ms" or excerpt["provenance"] != "local_speech_recognition" or excerpt["text"] != first["result"]["snippet"] or excerpt["source"] != source:
raise ValueError("Actual first recognized source citation missing")
ranked[query["id"]] = [{"asset": first["asset"], "start_ms": source["number"], "end_ms": excerpt["end_ms"]}]
diagnostic["recognized_speech_first"] = True
diagnostics.append(diagnostic)
scored = publisher_score(fixture, ranked) if eligible else None
file_top1 = sum(bool(row["hits"]) and row["hits"][0]["asset"] in row["query"]["expected"] for row in case["rows"])
return {"comparison_eligible": eligible, "strict_point_score": scored,
"correct_first_file": file_top1 if eligible else None,
"diagnostic_rows": diagnostics,
"scope": "Actual hybrid first result; raw media/non-speech origin remains an explicit recognized-speech timing failure; no lower hit replaces first; no carried-over full-source VAD score"}
def main():
parser = argparse.ArgumentParser()
parser.add_argument("report", type=Path)
parser.add_argument("fixture", type=Path)
parser.add_argument("policy", type=Path)
parser.add_argument("output", type=Path)
args = parser.parse_args()
raw = args.report.read_bytes()
fixture_raw = args.fixture.read_bytes()
policy_raw = args.policy.read_bytes()
report = json.loads(raw)
policy = json.loads(policy_raw)
if hashlib.sha256(fixture_raw).hexdigest() != FIXTURE_SHA or hashlib.sha256(policy_raw).hexdigest() != report["timing_policy_sha256"]:
raise ValueError("Frozen fixture/policy changed")
scorer_raw = Path(__file__).with_name("score.py").read_bytes()
if policy["scoring_function_sha256"] != hashlib.sha256(scorer_raw).hexdigest():
raise ValueError("Publisher scoring function changed")
result = evaluate(report, json.loads(fixture_raw))
result.update(raw_report_sha256=hashlib.sha256(raw).hexdigest(), fixture_sha256=FIXTURE_SHA, timing_policy_sha256=report["timing_policy_sha256"], scoring_function_sha256=policy["scoring_function_sha256"])
with args.output.open("x", encoding="utf-8") as output:
json.dump(result, output, ensure_ascii=False, indent=2)
if __name__ == "__main__":
main()
tools/speech-eval/test_score_service_timing.py
+52
View File
@@ -0,0 +1,52 @@
import copy
import unittest
from run import FIXTURE_SHA
from score_service_timing import evaluate, SOURCES
class TimingTests(unittest.TestCase):
def inputs(self):
queries = [{"id": "q" + str(index), "text": "unchanged", "category": "video-speech", "expected": ["clip04"], "intervals": {"clip04": [[1.551, 3.603]]}} for index in range(8)]
fixture = {"queries": queries}
sources = [{"id": id, "windows": [{"pcm_sha256": id}]} for id in SOURCES]
calls = [{"pcm_sha256": source["id"], "transcription": {"passages": []}} for source in sources]
source = {"kind": "timestamp_ms", "number": 1551}
hit = {"asset": "clip04", "result": {"match_kind": "semantic_speech", "snippet": "real", "source": source}, "verified_recognized_excerpt": {"provenance": "local_speech_recognition", "source": source, "end_ms": 3603, "text": "real"}}
rows = [{"query": query, "hits": [copy.deepcopy(hit)]} for query in queries]
return {"complete": True, "fixture_sha256": FIXTURE_SHA, "timing": {"sources": sources, "native_calls": calls, "comparison_eligible": True, "rows": rows}}, fixture
def test_frozen_seconds_map_to_actual_integer_milliseconds(self):
report, fixture = self.inputs()
result = evaluate(report, fixture)
self.assertEqual(result["strict_point_score"]["point_hits"], 8)
self.assertEqual(result["strict_point_score"]["overlap_hits"], 8)
def test_lower_timestamp_inside_label_is_never_selected(self):
report, fixture = self.inputs()
for row in report["timing"]["rows"]:
original = copy.deepcopy(row["hits"][0])
row["hits"][0]["result"]["source"]["number"] = 0
row["hits"][0]["verified_recognized_excerpt"]["source"]["number"] = 0
row["hits"].append(original)
self.assertEqual(evaluate(report, fixture)["strict_point_score"]["point_hits"], 0)
def test_visual_first_result_is_not_relabelled_speech(self):
report, fixture = self.inputs()
for row in report["timing"]["rows"]:
row["hits"][0]["result"]["match_kind"] = "semantic_video"
self.assertEqual(evaluate(report, fixture)["strict_point_score"]["point_hits"], 0)
def test_dropped_no_speech_competitor_invalidates_numeric_score(self):
report, fixture = self.inputs()
report["timing"]["native_calls"].pop()
result = evaluate(report, fixture)
self.assertFalse(result["comparison_eligible"])
self.assertIsNone(result["strict_point_score"])
def test_modified_query_is_rejected(self):
report, fixture = self.inputs()
report = copy.deepcopy(report)
report["timing"]["rows"][0]["query"]["text"] = "changed"
with self.assertRaises(ValueError):
evaluate(report, fixture)