Record real Google mixed retrieval baseline and remaining quality failures

docs/MIXED-QUALITY.md

@@ -1,32 +1,116 @@
-# Mixed retrieval quality gate
+# Mixed retrieval quality checkpoint
 
-A frozen mixed diagnostic is prepared in [fixtures/mixed](../fixtures/mixed/README.md).
+The frozen [mixed diagnostic](../fixtures/mixed/README.md) was executed on
-It addresses the missing comparison between natural-language All-files queries
+2026-10-13 with the genuine Google EmbeddingGemma 2 Q8_0 model, the normal
-and explicit Code scope while images, music, speech, videos and actual documents
+CPU/Vulkan worker forced to CPU, and the shared CLI/MCP engine. Windows 11 VM;
-compete. It contains 32 real files and 91 English/German queries, including two
+one inference thread. The CLI was a debug build. The corpus contains 32 actual
-pinned upstream llama.cpp implementations. The existing media labels are reused
+files: eight implementations (six searchi files and two pinned llama.cpp files),
-with their licenses and attribution. They are not an untouched held-out set.
+four technical documents, and twenty licensed media files. It is a small
+single-annotator diagnostic, with reused previously evaluated media labels;
+**these results do not establish production quality or untouched held-out quality**.
 
-**No actual mixed inference results are recorded yet.** The frozen manifest
+## Completed execution
-SHA-256 is `f83d27227582f614159ff153654d5bb5a47bf74998141e389557f3950ea0b924`.
-Asset/hash verification and six meaningful scoring/provenance tests passed.
-These prove harness behavior only; they establish no retrieval accuracy.
 
-The [evaluator](../tools/mixed-eval/README.md) calls the shared production engine
+The evaluator finished successfully with 210 rows: 104 executed requests per
-through CLI indexing and MCP search. It compares lexical/hybrid retrieval and
+mode plus one unsupported temporal-filter query per mode. All 32 files were
-All-files/Code scope without changing ranking or tuning a cutoff against these
+indexed; document extraction reported zero failures. Semantic indexing took
-labels. File-level recall and no-answer returned-result rates remain separate
+**1,060,840 ms (17 minutes 40.84 seconds)** and published generation one with
-from spoken timestamp localization. Raw evidence must preserve actual failures,
+**182 chunks: 64 text and 118 media**, derived from 62 document passages and
-encoding errors and unsupported filters.
+20 media files. The final report records zero media failures and an empty
+failure list. It made 62 document, 99 image/frame and 19 audio inference calls;
+82 source tasks were encoded and none reused. Image/frame calls plus audio calls
+account for the 118 media chunks. Indexing costs include decoder work and model
+initialization; they are not pure native inference measurements.
 
-Remaining work:
+| Positive queries | Hybrid All files top1 | Hybrid All files recall@5 |
+| --- | --- | --- |
+| Implementation intent | 10/14 | 13/14 |
+| Technical documents | 7/8 | 8/8 |
+| Images | 24/24 | 24/24 |
+| Videos, file identity only | 22/22 | 22/22 |
+| Music | 2/4 | 4/4 |
+| Sounds | 4/4 | 4/4 |
+| Exact media metadata/filter | 4/4 | 4/4 |
+| **All positive queries** | **73/80** | **79/80** |
 
-- Run the frozen corpus with the verified Google model and CPU runtime in a
+Lexical-only All files found the expected file first for **3/80** positives,
-  coordinated slot; record exact CLI/worker hashes and actual raw results.
+with **3/80** recall@5. In explicit Code scope, hybrid found **14/14** code
-- Inspect failures by modality, language, requested artifact type and scope;
+queries first; lexical-only found **0/14**. This covers two upstream files and
-  distinguish correct-file retrieval from a useful passage/timestamp.
+six self-repository implementations, not fourteen independent projects.
-- Author a larger independently reviewed held-out corpus before ranking changes;
+
-  frozen diagnostics must not become the sole optimization target.
+All **10/10 no-answer queries returned results** in hybrid All files;
-- Establish recall and no-answer acceptance targets on representative user
+lexical-only returned no results for those ten. This exposes uncalibrated
-  libraries, unrelated repositories and longer recordings. Tiny diagnostics
+irrelevant neighbors, rather than proving an answer exists. Do not interpret
-  cannot close the production relevance gate.
+the displayed hybrid scores as confidence or tune a rejection threshold on
+these same frozen labels. One query's requested temporal-range filter remains
+unsupported and was explicitly skipped in each mode.
+
+## Concrete failures
+
+- English `code that chooses audio search when a query mentions speech` returned
+  audio matches and omitted the expected routing implementation entirely from
+  twenty results. Explicit Code scope placed that implementation first. The
+  German paired request found it first in both scopes. This confirms a real
+  All-files audio-intent suppression failure; the separate routing audit
+  explains the current rule. No production ranking change was made for this run.
+- Both CPU-budget implementation queries ranked the performance documentation
+  above the implementation (rank two). The English rendering/queue timing
+  implementation query also placed the documentation first. Code scope resolved
+  these small cases, but requiring users to switch scope does not establish the
+  requested All-files experience.
+- The German media-change limitations query ranked incremental-indexing
+  documentation above the intended automatic-indexing document (rank two).
+  Its desired source remained present in the first five.
+- The English and German jazzy-violin queries ranked piano music first and
+  the correct violin music second. The frozen music hard negative remains a
+  genuine top-result failure.
+
+Video/speech rows above measure **correct file identity**, not a useful scene or
+spoken passage. They do not overturn the poor strict speech timestamp results
+in [MEDIA-QUALITY.md](MEDIA-QUALITY.md). Returned source timestamps are retained
+in the raw rows for further inspection. A correctly retrieved video can still
+open at the wrong moment.
+
+## Timing boundaries and evidence
+
+Across 90 executed All-files requests in their fixed fixture order, observed
+hybrid MCP roundtrip median/p95 were **101.98/136.38 ms**; native basic search
+median/p95 were **94.22/125.90 ms**. Lexical roundtrip median/p95 were
+**2.33/3.24 ms**, with native **1.31/2.14 ms**. The first hybrid request took
+117.73 ms roundtrip and 107.04 ms natively. These are ordered tiny-corpus
+observations, excluding process initialization/model loading; they are neither
+first user-input timing nor randomized production-scale warm p95. Code-scope
+requests alternate with All-files code queries, so task/filter caches and prior
+requests are part of this run's conditions.
+
+The complete raw report remains in ignored `.searchi/qa/mixed-v1.json` (636,101
+bytes), SHA-256 `c6619d4a66332f0378d42b9ec4f108349ce095aeb2dbcccd535de6095438b37a`.
+It retains every ordered hit, query, expected label, source location, native
+metric, index stdout/stderr and final semantic report. Runtime stderr retained
+Google token-type warnings and the native runtime's experimental audio warning;
+these were not hidden as failed assets or omitted from evidence.
+
+| Provenance | SHA-256 |
+| --- | --- |
+| Frozen labels | `f83d27227582f614159ff153654d5bb5a47bf74998141e389557f3950ea0b924` |
+| Actual debug CLI | `23afd9986d987bc45db9e08aa50f1070f3ad2229d275793c1f2a41ee0fcea545` |
+| Actual normal worker | `c01cb3ec13cfca722ee768e8350e1399a8b9a4d14a24a35adc3473a5cfc82a73` |
+| Pinned Google model manifest | `fad82f171f4976d8eeb89c11d6a79fc46d92afb73c10a34f140d5b56e377397f` |
+
+The worker/model pair was SHA-verified before inference. The active fingerprint
+records query context 512 and document context 2048. The owned evaluator session
+finished with exit zero and cleaned its temporary corpus/index and worker; no
+restart or substituted model was used. Reproduce with the
+[evaluator instructions](../tools/mixed-eval/README.md).
+
+## Open quality gates
+
+- Correct the documented All-files audio-intent suppression without hiding
+  legitimate media results; validate candidate preservation before ranking.
+- Evaluate general/code query task fusion on a new independently labelled set,
+  with bounded retrieval and measured cache/filter costs.
+- Improve music discrimination and precise speech timestamps.
+- Establish no-answer behavior on a separate calibration/held-out split while
+  preserving positive recall.
+- Add unrelated repositories, customer documents, longer recordings, human
+  listening and a second blinded annotator before broad relevance claims.

docs/ROADMAP.md

@@ -214,8 +214,11 @@ Completion requires recorded evidence; targets are not measured results.
   symbols, near-duplicate routines and queries with no relevant implementation.
   - [x] Audit mixed-task routing and eligibility cache behavior; see
     [routing audit](RETRIEVAL-ROUTING.md).
-  - [ ] Complete the frozen mixed baseline, then measure task fusion and cache
+  - [x] Complete the unchanged 32-file/91-query real Google CPU mixed baseline:
-    cost without tuning solely against the diagnostic labels.
+    Code scope 14/14 top-1 versus All files 10/14; all 10 no-answer queries
+    returned candidates. See [mixed quality](MIXED-QUALITY.md).
+  - [ ] Measure task fusion and cache cost against independently reviewed
+    held-out data, preserving the frozen diagnostic for regression comparison.
 - [ ] Validate code search UI interaction on native Windows/macOS/Linux and
   actual GPU hardware. Tiny CPU fixtures do not close those release gates.
 

fixtures/mixed/README.md

@@ -1,7 +1,8 @@
 # Mixed All-files diagnostic v1
 
-This is a frozen, small diagnostic, **not a production quality score**. No mixed
+This is a frozen, small diagnostic, **not a production quality score**. The first genuine CPU mixed
-model run has been performed yet.
+run is recorded in [MIXED-QUALITY.md](../../docs/MIXED-QUALITY.md); its failures
+remain visible and v1 labels remain unchanged.
 
 The corpus contains 32 actual files: six searchi implementations, four searchi
 technical documents, two complete pinned llama.cpp source files, and twenty