From f48a607bfaecf8775ad52c63e1880cc7b0b0b07e Mon Sep 17 00:00:00 2001 From: karl Date: Tue, 13 Oct 2026 17:49:10 +0200 Subject: [PATCH] Record full Google CPU routing regression with independently verified scores --- docs/MEDIA-QUALITY.md | 8 +++++ docs/MIXED-QUALITY-0.1.4.md | 64 +++++++++++++++++++++++++++++++++++++ docs/MIXED-QUALITY.md | 4 +++ docs/RETRIEVAL-ROUTING.md | 10 ++++-- docs/ROADMAP.md | 2 +- 5 files changed, 84 insertions(+), 4 deletions(-) create mode 100644 docs/MIXED-QUALITY-0.1.4.md diff --git a/docs/MEDIA-QUALITY.md b/docs/MEDIA-QUALITY.md index 8901ce0..ec2766f 100644 --- a/docs/MEDIA-QUALITY.md +++ b/docs/MEDIA-QUALITY.md @@ -90,6 +90,14 @@ The existing **20-document/24-query synthetic regression** was rerun with the ac Measured hybrid p95 was **146.83 ms** on this run. [Recorded output](../fixtures/media/document-baseline.json). This remains a synthetic seed, not independent production-quality evidence or a large-corpus performance claim. +## Current mixed regression follow-up + +The completed [0.1.4 mixed regression](MIXED-QUALITY-0.1.4.md) retains the known +file-level media outcomes with no scored regression: image top-1 24/24, video +file identity 22/22, music 2/4 and sound 4/4. Its one improvement concerns a +code/audio routing query. It does not rescore precise speech timestamps or add +independent media labels, so the experimental media rating remains 4/10. + ## Scoring and release gates ### Local recognized-speech supplement: actual Google diagnostic diff --git a/docs/MIXED-QUALITY-0.1.4.md b/docs/MIXED-QUALITY-0.1.4.md new file mode 100644 index 0000000..441cbee --- /dev/null +++ b/docs/MIXED-QUALITY-0.1.4.md @@ -0,0 +1,64 @@ +# Mixed retrieval regression on the 0.1.4 engine + +The unchanged frozen 32-file/91-query diagnostic completed on Windows 11 VM +using actual Google EmbeddingGemma 2, the normal runtime forced to CPU, and one +inference thread. The release CLI was built from shared engine source `ac90ce7`; +the earlier baseline used a debug CLI. No labels, rejection threshold, model, +worker or ranking parameters were tuned from this rerun. + +| Hybrid retrieval | Preserved baseline | Current run | +| --- | ---: | ---: | +| All-files positive queries: correct first file | 73/80 | 74/80 | +| All-files positive queries: recall in first five | 79/80 | 80/80 | +| Code intents in All files: correct first file | 10/14 | 11/14 | +| Code intents in All files: recall in first five | 13/14 | 14/14 | +| Explicit Code scope: correct first file | 14/14 | 14/14 | +| No-answer queries returning candidates (failure) | 10/10 | 10/10 | +| Queries with labelled hard negatives in first five | 18 | 18 | + +Only one scored row changed: `code-audio-en`, asking for code choosing audio +search for speech queries, previously omitted its implementation from all 20 +results; the intended source is now rank one. Independent read-only review +recomputed all stored scores from ordered hits and labels: no scored regression +across 208 executed rows. There are 210 unique rows including two unsupported +temporal-filter requests, which remain unexecuted rather than counted as passes. + +Document top-1 remains 7/8, images 24/24, video file identity 22/22, music 2/4, +sounds 4/4 and exact metadata/filter positives 4/4. This fixes the known routing +failure without regression on this diagnostic. It does not establish general +relevance, precise playback timestamps or calibrated rejection of unrelated files. + +## Matching inputs and execution + +Fixture/model-manifest/worker hashes and model fingerprint match the preserved +baseline. Both runs indexed 32 files with zero extraction/media failures, +62 passages, 182 chunks and identical 62 document, 99 image/frame and 19 audio +inference calls. Neither reused completed source tasks. The fixture and media +labels were previously examined; this is not untouched held-out evidence. + +Indexing took 2.33 s lexical and 1018.50 s semantic. Across 90 fixed-order hybrid +All-files requests, MCP roundtrip median/p95 were 96.27/119.15 ms and native +basic-search median/p95 were 94.14/117.08 ms. First hybrid +request was 113.39 ms roundtrip and 109.81 ms native. These exclude initialization, +use retained sessions/caches, and are not first desktop input or large-library +benchmarks. Different CLI build profiles mean timing differences are not an +isolated measurement of the routing change. + +Ignored raw report `.searchi/qa/mixed-v1-current-014.json`: 638814 bytes, SHA-256 +`b45e096f6f73c7f9669a129384f8b218edceb079891c069e02eb967ab118c17d`. +Preserved baseline SHA-256 remains +`c6619d4a66332f0378d42b9ec4f108349ce095aeb2dbcccd535de6095438b37a`. +Current CLI SHA-256 is +`e473a59ba0d027263be196cf13fcd764723cad0ecbe077d2e49344d32dde92a6`. +Normal worker SHA-256 is +`c01cb3ec13cfca722ee768e8350e1399a8b9a4d14a24a35adc3473a5cfc82a73`. +The report contains indexing diagnostics; this evaluator does not retain the +separate MCP session stderr stream. No complete MCP-warning capture is claimed. + +## What still matters + +Independent repositories/customer files, German speech, music hard negatives, +speech localization, no-answer calibration and native UI/GPU checks remain open. +The experimental media rating remains 4/10 under [the existing rubric](MEDIA-QUALITY.md). +The failed earlier held-out cutoff must not be retuned on these known results. +Continue with independent calibration evidence before changing retrieval policy. diff --git a/docs/MIXED-QUALITY.md b/docs/MIXED-QUALITY.md index 7c06262..55e841a 100644 --- a/docs/MIXED-QUALITY.md +++ b/docs/MIXED-QUALITY.md @@ -1,5 +1,9 @@ # Mixed retrieval quality checkpoint +This page preserves the pre-routing-fix baseline. The completed +[0.1.4 regression comparison](MIXED-QUALITY-0.1.4.md) fixes one known code/audio +routing failure with no scored regression; broader quality gates remain open. + The frozen [mixed diagnostic](../fixtures/mixed/README.md) was executed on 2026-10-13 with the genuine Google EmbeddingGemma 2 Q8_0 model, the normal CPU/Vulkan worker forced to CPU, and the shared CLI/MCP engine. Windows 11 VM; diff --git a/docs/RETRIEVAL-ROUTING.md b/docs/RETRIEVAL-ROUTING.md index c1005e9..70046cd 100644 --- a/docs/RETRIEVAL-ROUTING.md +++ b/docs/RETRIEVAL-ROUTING.md @@ -1,10 +1,10 @@ # Mixed query routing audit Read-only implementation audit, 2026-10-13, source revision `5d9cfc4`. -These findings describe current behavior; they are not a retrieval-quality +These opening findings describe that pre-fix snapshot; they are not a retrieval-quality measurement or an implemented ranking change. -## Current behavior +## Pre-fix behavior at 5d9cfc4 `HybridSearch::query_vector` selects Google's CodeRetrieval task only when `SearchOptions::code_only` is true. All-files searches use the general task, @@ -14,7 +14,7 @@ accidentally. See `crates/searchi-service/src/lib.rs`. The audio preference in `crates/searchi-service/src/routing.rs` recognizes words such as speech, sound and music. Document and image cues veto that preference; -code cues currently do not. If the preferred audio search returns any qualifying +code cues did not at this snapshot. If the preferred audio search returns any qualifying candidate, it becomes the exclusive semantic lane. Consequently a query such as "code that detects speech" can exclude semantic code candidates in All files. Lexical candidates still participate in hybrid fusion, and explicit Code scope @@ -72,6 +72,10 @@ diagnostic lane. Updated default semantic routing also returned the source implementation, proving the exclusive suppression is removed in this fixture. The test asserts candidate preservation, not top-1 superiority or broad quality. The full mixed baseline above remains the unchanged pre-fix measurement. +The completed [0.1.4 full regression](MIXED-QUALITY-0.1.4.md) now confirms that +this known failed query returns the implementation first. Code intents in All +files improved from 10/14 to 11/14 top-1 and 13/14 to 14/14 recall@5, without +scored regression elsewhere. Independent relevance and no-answer gates stay open. With this fix and the verified-asset callback, the Windows CPU workspace suite passed 130 tests (23 explicitly ignored integration/resource checks), desktop diff --git a/docs/ROADMAP.md b/docs/ROADMAP.md index 90a8d23..d3b868d 100644 --- a/docs/ROADMAP.md +++ b/docs/ROADMAP.md @@ -345,7 +345,7 @@ build was inferred from that lookup; macOS execution gates remain unchecked. - [x] Publish Windows v0.1.4 with the timestamp correction; verify public bytes, packaged helper provenance and isolated real Google CPU query-worker startup. See [evidence](WINDOWS-0.1.4.md). - [x] Publish updated Linux headless and Debian packages with the timestamp correction; genuine archived-document CPU search, unchanged rescan and packaged query-worker startup passed. Public bytes verified; [evidence](LINUX-0.1.4.md). - [x] Protect mixed quality evidence from accidental report overwrite and partial JSON publication; eight evaluator contract tests passed. -- [ ] Finish the unchanged frozen mixed regression on the current 0.1.4 engine, then compare routing effects without changing labels or tuning against results. Active CPU-only report: `.searchi/qa/mixed-v1-current-014.json`; completion must be verified before reporting scores. +- [x] Finish the unchanged frozen mixed regression on the current 0.1.4 engine; independent score recomputation confirms one known routing improvement and no scored regression. All-files positive top-1 74/80, recall@5 80/80; no-answer failures remain 10/10. See [comparison](MIXED-QUALITY-0.1.4.md); independent quality gates stay open. Initial Windows delivery is complete. Native interaction/upgrade, actual GPU, macOS execution and independent media-quality work remain real follow-up; the