65 lines
3.6 KiB
Markdown
65 lines
3.6 KiB
Markdown
# Mixed retrieval regression on the 0.1.4 engine
|
|
|
|
The unchanged frozen 32-file/91-query diagnostic completed on Windows 11 VM
|
|
using actual Google EmbeddingGemma 2, the normal runtime forced to CPU, and one
|
|
inference thread. The release CLI was built from shared engine source `ac90ce7`;
|
|
the earlier baseline used a debug CLI. No labels, rejection threshold, model,
|
|
worker or ranking parameters were tuned from this rerun.
|
|
|
|
| Hybrid retrieval | Preserved baseline | Current run |
|
|
| --- | ---: | ---: |
|
|
| All-files positive queries: correct first file | 73/80 | 74/80 |
|
|
| All-files positive queries: recall in first five | 79/80 | 80/80 |
|
|
| Code intents in All files: correct first file | 10/14 | 11/14 |
|
|
| Code intents in All files: recall in first five | 13/14 | 14/14 |
|
|
| Explicit Code scope: correct first file | 14/14 | 14/14 |
|
|
| No-answer queries returning candidates (failure) | 10/10 | 10/10 |
|
|
| Queries with labelled hard negatives in first five | 18 | 18 |
|
|
|
|
Only one scored row changed: `code-audio-en`, asking for code choosing audio
|
|
search for speech queries, previously omitted its implementation from all 20
|
|
results; the intended source is now rank one. Independent read-only review
|
|
recomputed all stored scores from ordered hits and labels: no scored regression
|
|
across 208 executed rows. There are 210 unique rows including two unsupported
|
|
temporal-filter requests, which remain unexecuted rather than counted as passes.
|
|
|
|
Document top-1 remains 7/8, images 24/24, video file identity 22/22, music 2/4,
|
|
sounds 4/4 and exact metadata/filter positives 4/4. This fixes the known routing
|
|
failure without regression on this diagnostic. It does not establish general
|
|
relevance, precise playback timestamps or calibrated rejection of unrelated files.
|
|
|
|
## Matching inputs and execution
|
|
|
|
Fixture/model-manifest/worker hashes and model fingerprint match the preserved
|
|
baseline. Both runs indexed 32 files with zero extraction/media failures,
|
|
62 passages, 182 chunks and identical 62 document, 99 image/frame and 19 audio
|
|
inference calls. Neither reused completed source tasks. The fixture and media
|
|
labels were previously examined; this is not untouched held-out evidence.
|
|
|
|
Indexing took 2.33 s lexical and 1018.50 s semantic. Across 90 fixed-order hybrid
|
|
All-files requests, MCP roundtrip median/p95 were 96.27/119.15 ms and native
|
|
basic-search median/p95 were 94.14/117.08 ms. First hybrid
|
|
request was 113.39 ms roundtrip and 109.81 ms native. These exclude initialization,
|
|
use retained sessions/caches, and are not first desktop input or large-library
|
|
benchmarks. Different CLI build profiles mean timing differences are not an
|
|
isolated measurement of the routing change.
|
|
|
|
Ignored raw report `.searchi/qa/mixed-v1-current-014.json`: 638814 bytes, SHA-256
|
|
`b45e096f6f73c7f9669a129384f8b218edceb079891c069e02eb967ab118c17d`.
|
|
Preserved baseline SHA-256 remains
|
|
`c6619d4a66332f0378d42b9ec4f108349ce095aeb2dbcccd535de6095438b37a`.
|
|
Current CLI SHA-256 is
|
|
`e473a59ba0d027263be196cf13fcd764723cad0ecbe077d2e49344d32dde92a6`.
|
|
Normal worker SHA-256 is
|
|
`c01cb3ec13cfca722ee768e8350e1399a8b9a4d14a24a35adc3473a5cfc82a73`.
|
|
The report contains indexing diagnostics; this evaluator does not retain the
|
|
separate MCP session stderr stream. No complete MCP-warning capture is claimed.
|
|
|
|
## What still matters
|
|
|
|
Independent repositories/customer files, German speech, music hard negatives,
|
|
speech localization, no-answer calibration and native UI/GPU checks remain open.
|
|
The experimental media rating remains 4/10 under [the existing rubric](MEDIA-QUALITY.md).
|
|
The failed earlier held-out cutoff must not be retuned on these known results.
|
|
Continue with independent calibration evidence before changing retrieval policy.
|