searchi/docs/MIXED-QUALITY.md

Mixed retrieval quality checkpoint

This page preserves the pre-routing-fix baseline. The completed 0.1.4 regression comparison fixes one known code/audio routing failure with no scored regression; broader quality gates remain open.

The frozen mixed diagnostic was executed on 2026-10-13 with the genuine Google EmbeddingGemma 2 Q8_0 model, the normal CPU/Vulkan worker forced to CPU, and the shared CLI/MCP engine. Windows 11 VM; one inference thread. The CLI was a debug build. The corpus contains 32 actual files: eight implementations (six searchi files and two pinned llama.cpp files), four technical documents, and twenty licensed media files. It is a small single-annotator diagnostic, with reused previously evaluated media labels; these results do not establish production quality or untouched held-out quality.

Completed execution

The evaluator finished successfully with 210 rows: 104 executed requests per mode plus one unsupported temporal-filter query per mode. All 32 files were indexed; document extraction reported zero failures. Semantic indexing took 1,060,840 ms (17 minutes 40.84 seconds) and published generation one with 182 chunks: 64 text and 118 media, derived from 62 document passages and 20 media files. The final report records zero media failures and an empty failure list. It made 62 document, 99 image/frame and 19 audio inference calls; 82 source tasks were encoded and none reused. Image/frame calls plus audio calls account for the 118 media chunks. Indexing costs include decoder work and model initialization; they are not pure native inference measurements.

Positive queries	Hybrid All files top1	Hybrid All files recall@5
Implementation intent	10/14	13/14
Technical documents	7/8	8/8
Images	24/24	24/24
Videos, file identity only	22/22	22/22
Music	2/4	4/4
Sounds	4/4	4/4
Exact media metadata/filter	4/4	4/4
All positive queries	73/80	79/80

Lexical-only All files found the expected file first for 3/80 positives, with 3/80 recall@5. In explicit Code scope, hybrid found 14/14 code queries first; lexical-only found 0/14. This covers two upstream files and six self-repository implementations, not fourteen independent projects.

All 10/10 no-answer queries returned results in hybrid All files; lexical-only returned no results for those ten. This exposes uncalibrated irrelevant neighbors, rather than proving an answer exists. Do not interpret the displayed hybrid scores as confidence or tune a rejection threshold on these same frozen labels. One query's requested temporal-range filter remains unsupported and was explicitly skipped in each mode.

Concrete failures

• English code that chooses audio search when a query mentions speech returned audio matches and omitted the expected routing implementation entirely from twenty results. Explicit Code scope placed that implementation first. The German paired request found it first in both scopes. This confirms a real All-files audio-intent suppression failure; the separate routing audit explains the current rule. No production ranking change was made for this run.
• Both CPU-budget implementation queries ranked the performance documentation above the implementation (rank two). The English rendering/queue timing implementation query also placed the documentation first. Code scope resolved these small cases, but requiring users to switch scope does not establish the requested All-files experience.
• The German media-change limitations query ranked incremental-indexing documentation above the intended automatic-indexing document (rank two). Its desired source remained present in the first five.
• The English and German jazzy-violin queries ranked piano music first and the correct violin music second. The frozen music hard negative remains a genuine top-result failure.

Video/speech rows above measure correct file identity, not a useful scene or spoken passage. They do not overturn the poor strict speech timestamp results in MEDIA-QUALITY.md. Returned source timestamps are retained in the raw rows for further inspection. A correctly retrieved video can still open at the wrong moment.

Timing boundaries and evidence

Across 90 executed All-files requests in their fixed fixture order, observed hybrid MCP roundtrip median/p95 were 101.98/136.38 ms; native basic search median/p95 were 94.22/125.90 ms. Lexical roundtrip median/p95 were 2.33/3.24 ms, with native 1.31/2.14 ms. The first hybrid request took 117.73 ms roundtrip and 107.04 ms natively. These are ordered tiny-corpus observations, excluding process initialization/model loading; they are neither first user-input timing nor randomized production-scale warm p95. Code-scope requests alternate with All-files code queries, so task/filter caches and prior requests are part of this run's conditions.

The complete raw report remains in ignored .searchi/qa/mixed-v1.json (636,101 bytes), SHA-256 c6619d4a66332f0378d42b9ec4f108349ce095aeb2dbcccd535de6095438b37a. It retains every ordered hit, query, expected label, source location, native metric, index stdout/stderr and final semantic report. Runtime stderr retained Google token-type warnings and the native runtime's experimental audio warning; these were not hidden as failed assets or omitted from evidence.

Provenance	SHA-256
Frozen labels	f83d27227582f614159ff153654d5bb5a47bf74998141e389557f3950ea0b924
Actual debug CLI	23afd9986d987bc45db9e08aa50f1070f3ad2229d275793c1f2a41ee0fcea545
Actual normal worker	c01cb3ec13cfca722ee768e8350e1399a8b9a4d14a24a35adc3473a5cfc82a73
Pinned Google model manifest	fad82f171f4976d8eeb89c11d6a79fc46d92afb73c10a34f140d5b56e377397f

The worker/model pair was SHA-verified before inference. The active fingerprint records query context 512 and document context 2048. The owned evaluator session finished with exit zero and cleaned its temporary corpus/index and worker; no restart or substituted model was used. Reproduce with the evaluator instructions.

Open quality gates

• Validate the subsequent explicit-code routing fix on broader mixed queries without hiding legitimate media results. A tiny real-model candidate-preservation regression passed; see routing evidence. The frozen baseline recorded here remains the pre-fix measurement.
• Evaluate general/code query task fusion on a new independently labelled set, with bounded retrieval and measured cache/filter costs.
• Improve music discrimination and precise speech timestamps.
• Establish no-answer behavior on a separate calibration/held-out split while preserving positive recall.
• Add unrelated repositories, customer documents, longer recordings, human listening and a second blinded annotator before broad relevance claims.