diff --git a/docs/MIXED-QUALITY.md b/docs/MIXED-QUALITY.md index 97be64b..e6e2266 100644 --- a/docs/MIXED-QUALITY.md +++ b/docs/MIXED-QUALITY.md @@ -1,32 +1,116 @@ -# Mixed retrieval quality gate +# Mixed retrieval quality checkpoint -A frozen mixed diagnostic is prepared in [fixtures/mixed](../fixtures/mixed/README.md). -It addresses the missing comparison between natural-language All-files queries -and explicit Code scope while images, music, speech, videos and actual documents -compete. It contains 32 real files and 91 English/German queries, including two -pinned upstream llama.cpp implementations. The existing media labels are reused -with their licenses and attribution. They are not an untouched held-out set. +The frozen [mixed diagnostic](../fixtures/mixed/README.md) was executed on +2026-10-13 with the genuine Google EmbeddingGemma 2 Q8_0 model, the normal +CPU/Vulkan worker forced to CPU, and the shared CLI/MCP engine. Windows 11 VM; +one inference thread. The CLI was a debug build. The corpus contains 32 actual +files: eight implementations (six searchi files and two pinned llama.cpp files), +four technical documents, and twenty licensed media files. It is a small +single-annotator diagnostic, with reused previously evaluated media labels; +**these results do not establish production quality or untouched held-out quality**. -**No actual mixed inference results are recorded yet.** The frozen manifest -SHA-256 is `f83d27227582f614159ff153654d5bb5a47bf74998141e389557f3950ea0b924`. -Asset/hash verification and six meaningful scoring/provenance tests passed. -These prove harness behavior only; they establish no retrieval accuracy. +## Completed execution -The [evaluator](../tools/mixed-eval/README.md) calls the shared production engine -through CLI indexing and MCP search. It compares lexical/hybrid retrieval and -All-files/Code scope without changing ranking or tuning a cutoff against these -labels. File-level recall and no-answer returned-result rates remain separate -from spoken timestamp localization. Raw evidence must preserve actual failures, -encoding errors and unsupported filters. +The evaluator finished successfully with 210 rows: 104 executed requests per +mode plus one unsupported temporal-filter query per mode. All 32 files were +indexed; document extraction reported zero failures. Semantic indexing took +**1,060,840 ms (17 minutes 40.84 seconds)** and published generation one with +**182 chunks: 64 text and 118 media**, derived from 62 document passages and +20 media files. The final report records zero media failures and an empty +failure list. It made 62 document, 99 image/frame and 19 audio inference calls; +82 source tasks were encoded and none reused. Image/frame calls plus audio calls +account for the 118 media chunks. Indexing costs include decoder work and model +initialization; they are not pure native inference measurements. -Remaining work: +| Positive queries | Hybrid All files top1 | Hybrid All files recall@5 | +| --- | --- | --- | +| Implementation intent | 10/14 | 13/14 | +| Technical documents | 7/8 | 8/8 | +| Images | 24/24 | 24/24 | +| Videos, file identity only | 22/22 | 22/22 | +| Music | 2/4 | 4/4 | +| Sounds | 4/4 | 4/4 | +| Exact media metadata/filter | 4/4 | 4/4 | +| **All positive queries** | **73/80** | **79/80** | -- Run the frozen corpus with the verified Google model and CPU runtime in a - coordinated slot; record exact CLI/worker hashes and actual raw results. -- Inspect failures by modality, language, requested artifact type and scope; - distinguish correct-file retrieval from a useful passage/timestamp. -- Author a larger independently reviewed held-out corpus before ranking changes; - frozen diagnostics must not become the sole optimization target. -- Establish recall and no-answer acceptance targets on representative user - libraries, unrelated repositories and longer recordings. Tiny diagnostics - cannot close the production relevance gate. +Lexical-only All files found the expected file first for **3/80** positives, +with **3/80** recall@5. In explicit Code scope, hybrid found **14/14** code +queries first; lexical-only found **0/14**. This covers two upstream files and +six self-repository implementations, not fourteen independent projects. + +All **10/10 no-answer queries returned results** in hybrid All files; +lexical-only returned no results for those ten. This exposes uncalibrated +irrelevant neighbors, rather than proving an answer exists. Do not interpret +the displayed hybrid scores as confidence or tune a rejection threshold on +these same frozen labels. One query's requested temporal-range filter remains +unsupported and was explicitly skipped in each mode. + +## Concrete failures + +- English `code that chooses audio search when a query mentions speech` returned + audio matches and omitted the expected routing implementation entirely from + twenty results. Explicit Code scope placed that implementation first. The + German paired request found it first in both scopes. This confirms a real + All-files audio-intent suppression failure; the separate routing audit + explains the current rule. No production ranking change was made for this run. +- Both CPU-budget implementation queries ranked the performance documentation + above the implementation (rank two). The English rendering/queue timing + implementation query also placed the documentation first. Code scope resolved + these small cases, but requiring users to switch scope does not establish the + requested All-files experience. +- The German media-change limitations query ranked incremental-indexing + documentation above the intended automatic-indexing document (rank two). + Its desired source remained present in the first five. +- The English and German jazzy-violin queries ranked piano music first and + the correct violin music second. The frozen music hard negative remains a + genuine top-result failure. + +Video/speech rows above measure **correct file identity**, not a useful scene or +spoken passage. They do not overturn the poor strict speech timestamp results +in [MEDIA-QUALITY.md](MEDIA-QUALITY.md). Returned source timestamps are retained +in the raw rows for further inspection. A correctly retrieved video can still +open at the wrong moment. + +## Timing boundaries and evidence + +Across 90 executed All-files requests in their fixed fixture order, observed +hybrid MCP roundtrip median/p95 were **101.98/136.38 ms**; native basic search +median/p95 were **94.22/125.90 ms**. Lexical roundtrip median/p95 were +**2.33/3.24 ms**, with native **1.31/2.14 ms**. The first hybrid request took +117.73 ms roundtrip and 107.04 ms natively. These are ordered tiny-corpus +observations, excluding process initialization/model loading; they are neither +first user-input timing nor randomized production-scale warm p95. Code-scope +requests alternate with All-files code queries, so task/filter caches and prior +requests are part of this run's conditions. + +The complete raw report remains in ignored `.searchi/qa/mixed-v1.json` (636,101 +bytes), SHA-256 `c6619d4a66332f0378d42b9ec4f108349ce095aeb2dbcccd535de6095438b37a`. +It retains every ordered hit, query, expected label, source location, native +metric, index stdout/stderr and final semantic report. Runtime stderr retained +Google token-type warnings and the native runtime's experimental audio warning; +these were not hidden as failed assets or omitted from evidence. + +| Provenance | SHA-256 | +| --- | --- | +| Frozen labels | `f83d27227582f614159ff153654d5bb5a47bf74998141e389557f3950ea0b924` | +| Actual debug CLI | `23afd9986d987bc45db9e08aa50f1070f3ad2229d275793c1f2a41ee0fcea545` | +| Actual normal worker | `c01cb3ec13cfca722ee768e8350e1399a8b9a4d14a24a35adc3473a5cfc82a73` | +| Pinned Google model manifest | `fad82f171f4976d8eeb89c11d6a79fc46d92afb73c10a34f140d5b56e377397f` | + +The worker/model pair was SHA-verified before inference. The active fingerprint +records query context 512 and document context 2048. The owned evaluator session +finished with exit zero and cleaned its temporary corpus/index and worker; no +restart or substituted model was used. Reproduce with the +[evaluator instructions](../tools/mixed-eval/README.md). + +## Open quality gates + +- Correct the documented All-files audio-intent suppression without hiding + legitimate media results; validate candidate preservation before ranking. +- Evaluate general/code query task fusion on a new independently labelled set, + with bounded retrieval and measured cache/filter costs. +- Improve music discrimination and precise speech timestamps. +- Establish no-answer behavior on a separate calibration/held-out split while + preserving positive recall. +- Add unrelated repositories, customer documents, longer recordings, human + listening and a second blinded annotator before broad relevance claims. diff --git a/docs/ROADMAP.md b/docs/ROADMAP.md index 09fc960..96c18ed 100644 --- a/docs/ROADMAP.md +++ b/docs/ROADMAP.md @@ -214,8 +214,11 @@ Completion requires recorded evidence; targets are not measured results. symbols, near-duplicate routines and queries with no relevant implementation. - [x] Audit mixed-task routing and eligibility cache behavior; see [routing audit](RETRIEVAL-ROUTING.md). - - [ ] Complete the frozen mixed baseline, then measure task fusion and cache - cost without tuning solely against the diagnostic labels. + - [x] Complete the unchanged 32-file/91-query real Google CPU mixed baseline: + Code scope 14/14 top-1 versus All files 10/14; all 10 no-answer queries + returned candidates. See [mixed quality](MIXED-QUALITY.md). + - [ ] Measure task fusion and cache cost against independently reviewed + held-out data, preserving the frozen diagnostic for regression comparison. - [ ] Validate code search UI interaction on native Windows/macOS/Linux and actual GPU hardware. Tiny CPU fixtures do not close those release gates. diff --git a/fixtures/mixed/README.md b/fixtures/mixed/README.md index bdd8ac8..c80aae2 100644 --- a/fixtures/mixed/README.md +++ b/fixtures/mixed/README.md @@ -1,7 +1,8 @@ # Mixed All-files diagnostic v1 -This is a frozen, small diagnostic, **not a production quality score**. No mixed -model run has been performed yet. +This is a frozen, small diagnostic, **not a production quality score**. The first genuine CPU mixed +run is recorded in [MIXED-QUALITY.md](../../docs/MIXED-QUALITY.md); its failures +remain visible and v1 labels remain unchanged. The corpus contains 32 actual files: six searchi implementations, four searchi technical documents, two complete pinned llama.cpp source files, and twenty