Record full Google CPU routing regression with independently verified scores

This commit is contained in:
karl committed 2026-10-13 17:49:10 +02:00
1 parent 0b6c958907
commit f48a607bfa
5 files changed
+84 -4

No files matched your search

+8
View File
@@ -90,6 +90,14 @@ The existing **20-document/24-query synthetic regression** was rerun with the ac
Measured hybrid p95 was **146.83 ms** on this run. [Recorded output](../fixtures/media/document-baseline.json). This remains a synthetic seed, not independent production-quality evidence or a large-corpus performance claim.
## Current mixed regression follow-up
The completed [0.1.4 mixed regression](MIXED-QUALITY-0.1.4.md) retains the known
file-level media outcomes with no scored regression: image top-1 24/24, video
file identity 22/22, music 2/4 and sound 4/4. Its one improvement concerns a
code/audio routing query. It does not rescore precise speech timestamps or add
independent media labels, so the experimental media rating remains 4/10.
## Scoring and release gates
### Local recognized-speech supplement: actual Google diagnostic
+64
View File
@@ -0,0 +1,64 @@
# Mixed retrieval regression on the 0.1.4 engine
The unchanged frozen 32-file/91-query diagnostic completed on Windows 11 VM
using actual Google EmbeddingGemma 2, the normal runtime forced to CPU, and one
inference thread. The release CLI was built from shared engine source `ac90ce7`;
the earlier baseline used a debug CLI. No labels, rejection threshold, model,
worker or ranking parameters were tuned from this rerun.
| Hybrid retrieval | Preserved baseline | Current run |
| --- | ---: | ---: |
| All-files positive queries: correct first file | 73/80 | 74/80 |
| All-files positive queries: recall in first five | 79/80 | 80/80 |
| Code intents in All files: correct first file | 10/14 | 11/14 |
| Code intents in All files: recall in first five | 13/14 | 14/14 |
| Explicit Code scope: correct first file | 14/14 | 14/14 |
| No-answer queries returning candidates (failure) | 10/10 | 10/10 |
| Queries with labelled hard negatives in first five | 18 | 18 |
Only one scored row changed: `code-audio-en`, asking for code choosing audio
search for speech queries, previously omitted its implementation from all 20
results; the intended source is now rank one. Independent read-only review
recomputed all stored scores from ordered hits and labels: no scored regression
across 208 executed rows. There are 210 unique rows including two unsupported
temporal-filter requests, which remain unexecuted rather than counted as passes.
Document top-1 remains 7/8, images 24/24, video file identity 22/22, music 2/4,
sounds 4/4 and exact metadata/filter positives 4/4. This fixes the known routing
failure without regression on this diagnostic. It does not establish general
relevance, precise playback timestamps or calibrated rejection of unrelated files.
## Matching inputs and execution
Fixture/model-manifest/worker hashes and model fingerprint match the preserved
baseline. Both runs indexed 32 files with zero extraction/media failures,
62 passages, 182 chunks and identical 62 document, 99 image/frame and 19 audio
inference calls. Neither reused completed source tasks. The fixture and media
labels were previously examined; this is not untouched held-out evidence.
Indexing took 2.33 s lexical and 1018.50 s semantic. Across 90 fixed-order hybrid
All-files requests, MCP roundtrip median/p95 were 96.27/119.15 ms and native
basic-search median/p95 were 94.14/117.08 ms. First hybrid
request was 113.39 ms roundtrip and 109.81 ms native. These exclude initialization,
use retained sessions/caches, and are not first desktop input or large-library
benchmarks. Different CLI build profiles mean timing differences are not an
isolated measurement of the routing change.
Ignored raw report `.searchi/qa/mixed-v1-current-014.json`: 638814 bytes, SHA-256
`b45e096f6f73c7f9669a129384f8b218edceb079891c069e02eb967ab118c17d`.
Preserved baseline SHA-256 remains
`c6619d4a66332f0378d42b9ec4f108349ce095aeb2dbcccd535de6095438b37a`.
Current CLI SHA-256 is
`e473a59ba0d027263be196cf13fcd764723cad0ecbe077d2e49344d32dde92a6`.
Normal worker SHA-256 is
`c01cb3ec13cfca722ee768e8350e1399a8b9a4d14a24a35adc3473a5cfc82a73`.
The report contains indexing diagnostics; this evaluator does not retain the
separate MCP session stderr stream. No complete MCP-warning capture is claimed.
## What still matters
Independent repositories/customer files, German speech, music hard negatives,
speech localization, no-answer calibration and native UI/GPU checks remain open.
The experimental media rating remains 4/10 under [the existing rubric](MEDIA-QUALITY.md).
The failed earlier held-out cutoff must not be retuned on these known results.
Continue with independent calibration evidence before changing retrieval policy.
+4
View File
@@ -1,5 +1,9 @@
# Mixed retrieval quality checkpoint
This page preserves the pre-routing-fix baseline. The completed
[0.1.4 regression comparison](MIXED-QUALITY-0.1.4.md) fixes one known code/audio
routing failure with no scored regression; broader quality gates remain open.
The frozen [mixed diagnostic](../fixtures/mixed/README.md) was executed on
2026-10-13 with the genuine Google EmbeddingGemma 2 Q8_0 model, the normal
CPU/Vulkan worker forced to CPU, and the shared CLI/MCP engine. Windows 11 VM;
+7 -3
View File
@@ -1,10 +1,10 @@
# Mixed query routing audit
Read-only implementation audit, 2026-10-13, source revision `5d9cfc4`.
These findings describe current behavior; they are not a retrieval-quality
These opening findings describe that pre-fix snapshot; they are not a retrieval-quality
measurement or an implemented ranking change.
## Current behavior
## Pre-fix behavior at 5d9cfc4
`HybridSearch::query_vector` selects Google's CodeRetrieval task only when
`SearchOptions::code_only` is true. All-files searches use the general task,
@@ -14,7 +14,7 @@ accidentally. See `crates/searchi-service/src/lib.rs`.
The audio preference in `crates/searchi-service/src/routing.rs` recognizes words
such as speech, sound and music. Document and image cues veto that preference;
code cues currently do not. If the preferred audio search returns any qualifying
code cues did not at this snapshot. If the preferred audio search returns any qualifying
candidate, it becomes the exclusive semantic lane. Consequently a query such as
"code that detects speech" can exclude semantic code candidates in All files.
Lexical candidates still participate in hybrid fusion, and explicit Code scope
@@ -72,6 +72,10 @@ diagnostic lane. Updated default semantic routing also returned the source
implementation, proving the exclusive suppression is removed in this fixture.
The test asserts candidate preservation, not top-1 superiority or broad quality.
The full mixed baseline above remains the unchanged pre-fix measurement.
The completed [0.1.4 full regression](MIXED-QUALITY-0.1.4.md) now confirms that
this known failed query returns the implementation first. Code intents in All
files improved from 10/14 to 11/14 top-1 and 13/14 to 14/14 recall@5, without
scored regression elsewhere. Independent relevance and no-answer gates stay open.
With this fix and the verified-asset callback, the Windows CPU workspace suite
passed 130 tests (23 explicitly ignored integration/resource checks), desktop
+1 -1
View File
@@ -345,7 +345,7 @@ build was inferred from that lookup; macOS execution gates remain unchecked.
- [x] Publish Windows v0.1.4 with the timestamp correction; verify public bytes, packaged helper provenance and isolated real Google CPU query-worker startup. See [evidence](WINDOWS-0.1.4.md).
- [x] Publish updated Linux headless and Debian packages with the timestamp correction; genuine archived-document CPU search, unchanged rescan and packaged query-worker startup passed. Public bytes verified; [evidence](LINUX-0.1.4.md).
- [x] Protect mixed quality evidence from accidental report overwrite and partial JSON publication; eight evaluator contract tests passed.
- [ ] Finish the unchanged frozen mixed regression on the current 0.1.4 engine, then compare routing effects without changing labels or tuning against results. Active CPU-only report: `.searchi/qa/mixed-v1-current-014.json`; completion must be verified before reporting scores.
- [x] Finish the unchanged frozen mixed regression on the current 0.1.4 engine; independent score recomputation confirms one known routing improvement and no scored regression. All-files positive top-1 74/80, recall@5 80/80; no-answer failures remain 10/10. See [comparison](MIXED-QUALITY-0.1.4.md); independent quality gates stay open.
Initial Windows delivery is complete. Native interaction/upgrade, actual GPU,
macOS execution and independent media-quality work remain real follow-up; the