Record full Google CPU routing regression with independently verified scores

This commit is contained in:
karl committed 2026-10-13 17:49:10 +02:00
1 parent 0b6c958907
commit f48a607bfa
5 files changed
+84 -4

No files matched your search

+64
View File
@@ -0,0 +1,64 @@
# Mixed retrieval regression on the 0.1.4 engine
The unchanged frozen 32-file/91-query diagnostic completed on Windows 11 VM
using actual Google EmbeddingGemma 2, the normal runtime forced to CPU, and one
inference thread. The release CLI was built from shared engine source `ac90ce7`;
the earlier baseline used a debug CLI. No labels, rejection threshold, model,
worker or ranking parameters were tuned from this rerun.
| Hybrid retrieval | Preserved baseline | Current run |
| --- | ---: | ---: |
| All-files positive queries: correct first file | 73/80 | 74/80 |
| All-files positive queries: recall in first five | 79/80 | 80/80 |
| Code intents in All files: correct first file | 10/14 | 11/14 |
| Code intents in All files: recall in first five | 13/14 | 14/14 |
| Explicit Code scope: correct first file | 14/14 | 14/14 |
| No-answer queries returning candidates (failure) | 10/10 | 10/10 |
| Queries with labelled hard negatives in first five | 18 | 18 |
Only one scored row changed: `code-audio-en`, asking for code choosing audio
search for speech queries, previously omitted its implementation from all 20
results; the intended source is now rank one. Independent read-only review
recomputed all stored scores from ordered hits and labels: no scored regression
across 208 executed rows. There are 210 unique rows including two unsupported
temporal-filter requests, which remain unexecuted rather than counted as passes.
Document top-1 remains 7/8, images 24/24, video file identity 22/22, music 2/4,
sounds 4/4 and exact metadata/filter positives 4/4. This fixes the known routing
failure without regression on this diagnostic. It does not establish general
relevance, precise playback timestamps or calibrated rejection of unrelated files.
## Matching inputs and execution
Fixture/model-manifest/worker hashes and model fingerprint match the preserved
baseline. Both runs indexed 32 files with zero extraction/media failures,
62 passages, 182 chunks and identical 62 document, 99 image/frame and 19 audio
inference calls. Neither reused completed source tasks. The fixture and media
labels were previously examined; this is not untouched held-out evidence.
Indexing took 2.33 s lexical and 1018.50 s semantic. Across 90 fixed-order hybrid
All-files requests, MCP roundtrip median/p95 were 96.27/119.15 ms and native
basic-search median/p95 were 94.14/117.08 ms. First hybrid
request was 113.39 ms roundtrip and 109.81 ms native. These exclude initialization,
use retained sessions/caches, and are not first desktop input or large-library
benchmarks. Different CLI build profiles mean timing differences are not an
isolated measurement of the routing change.
Ignored raw report `.searchi/qa/mixed-v1-current-014.json`: 638814 bytes, SHA-256
`b45e096f6f73c7f9669a129384f8b218edceb079891c069e02eb967ab118c17d`.
Preserved baseline SHA-256 remains
`c6619d4a66332f0378d42b9ec4f108349ce095aeb2dbcccd535de6095438b37a`.
Current CLI SHA-256 is
`e473a59ba0d027263be196cf13fcd764723cad0ecbe077d2e49344d32dde92a6`.
Normal worker SHA-256 is
`c01cb3ec13cfca722ee768e8350e1399a8b9a4d14a24a35adc3473a5cfc82a73`.
The report contains indexing diagnostics; this evaluator does not retain the
separate MCP session stderr stream. No complete MCP-warning capture is claimed.
## What still matters
Independent repositories/customer files, German speech, music hard negatives,
speech localization, no-answer calibration and native UI/GPU checks remain open.
The experimental media rating remains 4/10 under [the existing rubric](MEDIA-QUALITY.md).
The failed earlier held-out cutoff must not be retuned on these known results.
Continue with independent calibration evidence before changing retrieval policy.