Record full Google CPU routing regression with independently verified scores
This commit is contained in:
1 parent
0b6c958907
commit
f48a607bfa
5 files changed
+84
-4
No files matched your search
@@ -90,6 +90,14 @@ The existing **20-document/24-query synthetic regression** was rerun with the ac
|
||||
|
||||
Measured hybrid p95 was **146.83 ms** on this run. [Recorded output](../fixtures/media/document-baseline.json). This remains a synthetic seed, not independent production-quality evidence or a large-corpus performance claim.
|
||||
|
||||
## Current mixed regression follow-up
|
||||
|
||||
The completed [0.1.4 mixed regression](MIXED-QUALITY-0.1.4.md) retains the known
|
||||
file-level media outcomes with no scored regression: image top-1 24/24, video
|
||||
file identity 22/22, music 2/4 and sound 4/4. Its one improvement concerns a
|
||||
code/audio routing query. It does not rescore precise speech timestamps or add
|
||||
independent media labels, so the experimental media rating remains 4/10.
|
||||
|
||||
## Scoring and release gates
|
||||
|
||||
### Local recognized-speech supplement: actual Google diagnostic
|
||||
|
||||
@@ -0,0 +1,64 @@
|
||||
# Mixed retrieval regression on the 0.1.4 engine
|
||||
|
||||
The unchanged frozen 32-file/91-query diagnostic completed on Windows 11 VM
|
||||
using actual Google EmbeddingGemma 2, the normal runtime forced to CPU, and one
|
||||
inference thread. The release CLI was built from shared engine source `ac90ce7`;
|
||||
the earlier baseline used a debug CLI. No labels, rejection threshold, model,
|
||||
worker or ranking parameters were tuned from this rerun.
|
||||
|
||||
| Hybrid retrieval | Preserved baseline | Current run |
|
||||
| --- | ---: | ---: |
|
||||
| All-files positive queries: correct first file | 73/80 | 74/80 |
|
||||
| All-files positive queries: recall in first five | 79/80 | 80/80 |
|
||||
| Code intents in All files: correct first file | 10/14 | 11/14 |
|
||||
| Code intents in All files: recall in first five | 13/14 | 14/14 |
|
||||
| Explicit Code scope: correct first file | 14/14 | 14/14 |
|
||||
| No-answer queries returning candidates (failure) | 10/10 | 10/10 |
|
||||
| Queries with labelled hard negatives in first five | 18 | 18 |
|
||||
|
||||
Only one scored row changed: `code-audio-en`, asking for code choosing audio
|
||||
search for speech queries, previously omitted its implementation from all 20
|
||||
results; the intended source is now rank one. Independent read-only review
|
||||
recomputed all stored scores from ordered hits and labels: no scored regression
|
||||
across 208 executed rows. There are 210 unique rows including two unsupported
|
||||
temporal-filter requests, which remain unexecuted rather than counted as passes.
|
||||
|
||||
Document top-1 remains 7/8, images 24/24, video file identity 22/22, music 2/4,
|
||||
sounds 4/4 and exact metadata/filter positives 4/4. This fixes the known routing
|
||||
failure without regression on this diagnostic. It does not establish general
|
||||
relevance, precise playback timestamps or calibrated rejection of unrelated files.
|
||||
|
||||
## Matching inputs and execution
|
||||
|
||||
Fixture/model-manifest/worker hashes and model fingerprint match the preserved
|
||||
baseline. Both runs indexed 32 files with zero extraction/media failures,
|
||||
62 passages, 182 chunks and identical 62 document, 99 image/frame and 19 audio
|
||||
inference calls. Neither reused completed source tasks. The fixture and media
|
||||
labels were previously examined; this is not untouched held-out evidence.
|
||||
|
||||
Indexing took 2.33 s lexical and 1018.50 s semantic. Across 90 fixed-order hybrid
|
||||
All-files requests, MCP roundtrip median/p95 were 96.27/119.15 ms and native
|
||||
basic-search median/p95 were 94.14/117.08 ms. First hybrid
|
||||
request was 113.39 ms roundtrip and 109.81 ms native. These exclude initialization,
|
||||
use retained sessions/caches, and are not first desktop input or large-library
|
||||
benchmarks. Different CLI build profiles mean timing differences are not an
|
||||
isolated measurement of the routing change.
|
||||
|
||||
Ignored raw report `.searchi/qa/mixed-v1-current-014.json`: 638814 bytes, SHA-256
|
||||
`b45e096f6f73c7f9669a129384f8b218edceb079891c069e02eb967ab118c17d`.
|
||||
Preserved baseline SHA-256 remains
|
||||
`c6619d4a66332f0378d42b9ec4f108349ce095aeb2dbcccd535de6095438b37a`.
|
||||
Current CLI SHA-256 is
|
||||
`e473a59ba0d027263be196cf13fcd764723cad0ecbe077d2e49344d32dde92a6`.
|
||||
Normal worker SHA-256 is
|
||||
`c01cb3ec13cfca722ee768e8350e1399a8b9a4d14a24a35adc3473a5cfc82a73`.
|
||||
The report contains indexing diagnostics; this evaluator does not retain the
|
||||
separate MCP session stderr stream. No complete MCP-warning capture is claimed.
|
||||
|
||||
## What still matters
|
||||
|
||||
Independent repositories/customer files, German speech, music hard negatives,
|
||||
speech localization, no-answer calibration and native UI/GPU checks remain open.
|
||||
The experimental media rating remains 4/10 under [the existing rubric](MEDIA-QUALITY.md).
|
||||
The failed earlier held-out cutoff must not be retuned on these known results.
|
||||
Continue with independent calibration evidence before changing retrieval policy.
|
||||
@@ -1,5 +1,9 @@
|
||||
# Mixed retrieval quality checkpoint
|
||||
|
||||
This page preserves the pre-routing-fix baseline. The completed
|
||||
[0.1.4 regression comparison](MIXED-QUALITY-0.1.4.md) fixes one known code/audio
|
||||
routing failure with no scored regression; broader quality gates remain open.
|
||||
|
||||
The frozen [mixed diagnostic](../fixtures/mixed/README.md) was executed on
|
||||
2026-10-13 with the genuine Google EmbeddingGemma 2 Q8_0 model, the normal
|
||||
CPU/Vulkan worker forced to CPU, and the shared CLI/MCP engine. Windows 11 VM;
|
||||
|
||||
@@ -1,10 +1,10 @@
|
||||
# Mixed query routing audit
|
||||
|
||||
Read-only implementation audit, 2026-10-13, source revision `5d9cfc4`.
|
||||
These findings describe current behavior; they are not a retrieval-quality
|
||||
These opening findings describe that pre-fix snapshot; they are not a retrieval-quality
|
||||
measurement or an implemented ranking change.
|
||||
|
||||
## Current behavior
|
||||
## Pre-fix behavior at 5d9cfc4
|
||||
|
||||
`HybridSearch::query_vector` selects Google's CodeRetrieval task only when
|
||||
`SearchOptions::code_only` is true. All-files searches use the general task,
|
||||
@@ -14,7 +14,7 @@ accidentally. See `crates/searchi-service/src/lib.rs`.
|
||||
|
||||
The audio preference in `crates/searchi-service/src/routing.rs` recognizes words
|
||||
such as speech, sound and music. Document and image cues veto that preference;
|
||||
code cues currently do not. If the preferred audio search returns any qualifying
|
||||
code cues did not at this snapshot. If the preferred audio search returns any qualifying
|
||||
candidate, it becomes the exclusive semantic lane. Consequently a query such as
|
||||
"code that detects speech" can exclude semantic code candidates in All files.
|
||||
Lexical candidates still participate in hybrid fusion, and explicit Code scope
|
||||
@@ -72,6 +72,10 @@ diagnostic lane. Updated default semantic routing also returned the source
|
||||
implementation, proving the exclusive suppression is removed in this fixture.
|
||||
The test asserts candidate preservation, not top-1 superiority or broad quality.
|
||||
The full mixed baseline above remains the unchanged pre-fix measurement.
|
||||
The completed [0.1.4 full regression](MIXED-QUALITY-0.1.4.md) now confirms that
|
||||
this known failed query returns the implementation first. Code intents in All
|
||||
files improved from 10/14 to 11/14 top-1 and 13/14 to 14/14 recall@5, without
|
||||
scored regression elsewhere. Independent relevance and no-answer gates stay open.
|
||||
|
||||
With this fix and the verified-asset callback, the Windows CPU workspace suite
|
||||
passed 130 tests (23 explicitly ignored integration/resource checks), desktop
|
||||
|
||||
+1
-1
@@ -345,7 +345,7 @@ build was inferred from that lookup; macOS execution gates remain unchecked.
|
||||
- [x] Publish Windows v0.1.4 with the timestamp correction; verify public bytes, packaged helper provenance and isolated real Google CPU query-worker startup. See [evidence](WINDOWS-0.1.4.md).
|
||||
- [x] Publish updated Linux headless and Debian packages with the timestamp correction; genuine archived-document CPU search, unchanged rescan and packaged query-worker startup passed. Public bytes verified; [evidence](LINUX-0.1.4.md).
|
||||
- [x] Protect mixed quality evidence from accidental report overwrite and partial JSON publication; eight evaluator contract tests passed.
|
||||
- [ ] Finish the unchanged frozen mixed regression on the current 0.1.4 engine, then compare routing effects without changing labels or tuning against results. Active CPU-only report: `.searchi/qa/mixed-v1-current-014.json`; completion must be verified before reporting scores.
|
||||
- [x] Finish the unchanged frozen mixed regression on the current 0.1.4 engine; independent score recomputation confirms one known routing improvement and no scored regression. All-files positive top-1 74/80, recall@5 80/80; no-answer failures remain 10/10. See [comparison](MIXED-QUALITY-0.1.4.md); independent quality gates stay open.
|
||||
|
||||
Initial Windows delivery is complete. Native interaction/upgrade, actual GPU,
|
||||
macOS execution and independent media-quality work remain real follow-up; the
|
||||
|
||||
Reference in new issue
Block a user