3.6 KiB
Mixed retrieval regression on the 0.1.4 engine
The unchanged frozen 32-file/91-query diagnostic completed on Windows 11 VM
using actual Google EmbeddingGemma 2, the normal runtime forced to CPU, and one
inference thread. The release CLI was built from shared engine source ac90ce7;
the earlier baseline used a debug CLI. No labels, rejection threshold, model,
worker or ranking parameters were tuned from this rerun.
| Hybrid retrieval | Preserved baseline | Current run |
|---|---|---|
| All-files positive queries: correct first file | 73/80 | 74/80 |
| All-files positive queries: recall in first five | 79/80 | 80/80 |
| Code intents in All files: correct first file | 10/14 | 11/14 |
| Code intents in All files: recall in first five | 13/14 | 14/14 |
| Explicit Code scope: correct first file | 14/14 | 14/14 |
| No-answer queries returning candidates (failure) | 10/10 | 10/10 |
| Queries with labelled hard negatives in first five | 18 | 18 |
Only one scored row changed: code-audio-en, asking for code choosing audio
search for speech queries, previously omitted its implementation from all 20
results; the intended source is now rank one. Independent read-only review
recomputed all stored scores from ordered hits and labels: no scored regression
across 208 executed rows. There are 210 unique rows including two unsupported
temporal-filter requests, which remain unexecuted rather than counted as passes.
Document top-1 remains 7/8, images 24/24, video file identity 22/22, music 2/4, sounds 4/4 and exact metadata/filter positives 4/4. This fixes the known routing failure without regression on this diagnostic. It does not establish general relevance, precise playback timestamps or calibrated rejection of unrelated files.
Matching inputs and execution
Fixture/model-manifest/worker hashes and model fingerprint match the preserved baseline. Both runs indexed 32 files with zero extraction/media failures, 62 passages, 182 chunks and identical 62 document, 99 image/frame and 19 audio inference calls. Neither reused completed source tasks. The fixture and media labels were previously examined; this is not untouched held-out evidence.
Indexing took 2.33 s lexical and 1018.50 s semantic. Across 90 fixed-order hybrid All-files requests, MCP roundtrip median/p95 were 96.27/119.15 ms and native basic-search median/p95 were 94.14/117.08 ms. First hybrid request was 113.39 ms roundtrip and 109.81 ms native. These exclude initialization, use retained sessions/caches, and are not first desktop input or large-library benchmarks. Different CLI build profiles mean timing differences are not an isolated measurement of the routing change.
Ignored raw report .searchi/qa/mixed-v1-current-014.json: 638814 bytes, SHA-256
b45e096f6f73c7f9669a129384f8b218edceb079891c069e02eb967ab118c17d.
Preserved baseline SHA-256 remains
c6619d4a66332f0378d42b9ec4f108349ce095aeb2dbcccd535de6095438b37a.
Current CLI SHA-256 is
e473a59ba0d027263be196cf13fcd764723cad0ecbe077d2e49344d32dde92a6.
Normal worker SHA-256 is
c01cb3ec13cfca722ee768e8350e1399a8b9a4d14a24a35adc3473a5cfc82a73.
The report contains indexing diagnostics; this evaluator does not retain the
separate MCP session stderr stream. No complete MCP-warning capture is claimed.
What still matters
Independent repositories/customer files, German speech, music hard negatives, speech localization, no-answer calibration and native UI/GPU checks remain open. The experimental media rating remains 4/10 under the existing rubric. The failed earlier held-out cutoff must not be retuned on these known results. Continue with independent calibration evidence before changing retrieval policy.