searchi/docs/MIXED-QUALITY.md

# Mixed retrieval quality checkpoint

The frozen [mixed diagnostic](../fixtures/mixed/README.md) was executed on
2026-10-13 with the genuine Google EmbeddingGemma 2 Q8_0 model, the normal
CPU/Vulkan worker forced to CPU, and the shared CLI/MCP engine. Windows 11 VM;
one inference thread. The CLI was a debug build. The corpus contains 32 actual
files: eight implementations (six searchi files and two pinned llama.cpp files),
four technical documents, and twenty licensed media files. It is a small
single-annotator diagnostic, with reused previously evaluated media labels;
**these results do not establish production quality or untouched held-out quality**.

## Completed execution

The evaluator finished successfully with 210 rows: 104 executed requests per
mode plus one unsupported temporal-filter query per mode. All 32 files were
indexed; document extraction reported zero failures. Semantic indexing took
**1,060,840 ms (17 minutes 40.84 seconds)** and published generation one with
**182 chunks: 64 text and 118 media**, derived from 62 document passages and
20 media files. The final report records zero media failures and an empty
failure list. It made 62 document, 99 image/frame and 19 audio inference calls;
82 source tasks were encoded and none reused. Image/frame calls plus audio calls
account for the 118 media chunks. Indexing costs include decoder work and model
initialization; they are not pure native inference measurements.

| Positive queries | Hybrid All files top1 | Hybrid All files recall@5 |
| --- | --- | --- |
| Implementation intent | 10/14 | 13/14 |
| Technical documents | 7/8 | 8/8 |
| Images | 24/24 | 24/24 |
| Videos, file identity only | 22/22 | 22/22 |
| Music | 2/4 | 4/4 |
| Sounds | 4/4 | 4/4 |
| Exact media metadata/filter | 4/4 | 4/4 |
| **All positive queries** | **73/80** | **79/80** |

Lexical-only All files found the expected file first for **3/80** positives,
with **3/80** recall@5. In explicit Code scope, hybrid found **14/14** code
queries first; lexical-only found **0/14**. This covers two upstream files and
six self-repository implementations, not fourteen independent projects.

All **10/10 no-answer queries returned results** in hybrid All files;
lexical-only returned no results for those ten. This exposes uncalibrated
irrelevant neighbors, rather than proving an answer exists. Do not interpret
the displayed hybrid scores as confidence or tune a rejection threshold on
these same frozen labels. One query's requested temporal-range filter remains
unsupported and was explicitly skipped in each mode.

## Concrete failures

- English `code that chooses audio search when a query mentions speech` returned
  audio matches and omitted the expected routing implementation entirely from
  twenty results. Explicit Code scope placed that implementation first. The
  German paired request found it first in both scopes. This confirms a real
  All-files audio-intent suppression failure; the separate routing audit
  explains the current rule. No production ranking change was made for this run.
- Both CPU-budget implementation queries ranked the performance documentation
  above the implementation (rank two). The English rendering/queue timing
  implementation query also placed the documentation first. Code scope resolved
  these small cases, but requiring users to switch scope does not establish the
  requested All-files experience.
- The German media-change limitations query ranked incremental-indexing
  documentation above the intended automatic-indexing document (rank two).
  Its desired source remained present in the first five.
- The English and German jazzy-violin queries ranked piano music first and
  the correct violin music second. The frozen music hard negative remains a
  genuine top-result failure.

Video/speech rows above measure **correct file identity**, not a useful scene or
spoken passage. They do not overturn the poor strict speech timestamp results
in [MEDIA-QUALITY.md](MEDIA-QUALITY.md). Returned source timestamps are retained
in the raw rows for further inspection. A correctly retrieved video can still
open at the wrong moment.

## Timing boundaries and evidence

Across 90 executed All-files requests in their fixed fixture order, observed
hybrid MCP roundtrip median/p95 were **101.98/136.38 ms**; native basic search
median/p95 were **94.22/125.90 ms**. Lexical roundtrip median/p95 were
**2.33/3.24 ms**, with native **1.31/2.14 ms**. The first hybrid request took
117.73 ms roundtrip and 107.04 ms natively. These are ordered tiny-corpus
observations, excluding process initialization/model loading; they are neither
first user-input timing nor randomized production-scale warm p95. Code-scope
requests alternate with All-files code queries, so task/filter caches and prior
requests are part of this run's conditions.

The complete raw report remains in ignored `.searchi/qa/mixed-v1.json` (636,101
bytes), SHA-256 `c6619d4a66332f0378d42b9ec4f108349ce095aeb2dbcccd535de6095438b37a`.
It retains every ordered hit, query, expected label, source location, native
metric, index stdout/stderr and final semantic report. Runtime stderr retained
Google token-type warnings and the native runtime's experimental audio warning;
these were not hidden as failed assets or omitted from evidence.

| Provenance | SHA-256 |
| --- | --- |
| Frozen labels | `f83d27227582f614159ff153654d5bb5a47bf74998141e389557f3950ea0b924` |
| Actual debug CLI | `23afd9986d987bc45db9e08aa50f1070f3ad2229d275793c1f2a41ee0fcea545` |
| Actual normal worker | `c01cb3ec13cfca722ee768e8350e1399a8b9a4d14a24a35adc3473a5cfc82a73` |
| Pinned Google model manifest | `fad82f171f4976d8eeb89c11d6a79fc46d92afb73c10a34f140d5b56e377397f` |

The worker/model pair was SHA-verified before inference. The active fingerprint
records query context 512 and document context 2048. The owned evaluator session
finished with exit zero and cleaned its temporary corpus/index and worker; no
restart or substituted model was used. Reproduce with the
[evaluator instructions](../tools/mixed-eval/README.md).

## Open quality gates

- Validate the subsequent explicit-code routing fix on broader mixed queries
  without hiding legitimate media results. A tiny real-model candidate-preservation
  regression passed; see [routing evidence](RETRIEVAL-ROUTING.md). The frozen
  baseline recorded here remains the pre-fix measurement.
- Evaluate general/code query task fusion on a new independently labelled set,
  with bounded retrieval and measured cache/filter costs.
- Improve music discrimination and precise speech timestamps.
- Establish no-answer behavior on a separate calibration/held-out split while
  preserving positive recall.
- Add unrelated repositories, customer documents, longer recordings, human
  listening and a second blinded annotator before broad relevance claims.