123 lines
6.9 KiB
Markdown
123 lines
6.9 KiB
Markdown
# Mixed retrieval quality checkpoint
|
|
|
|
This page preserves the pre-routing-fix baseline. The completed
|
|
[0.1.4 regression comparison](MIXED-QUALITY-0.1.4.md) fixes one known code/audio
|
|
routing failure with no scored regression; broader quality gates remain open.
|
|
|
|
The frozen [mixed diagnostic](../fixtures/mixed/README.md) was executed on
|
|
2026-10-13 with the genuine Google EmbeddingGemma 2 Q8_0 model, the normal
|
|
CPU/Vulkan worker forced to CPU, and the shared CLI/MCP engine. Windows 11 VM;
|
|
one inference thread. The CLI was a debug build. The corpus contains 32 actual
|
|
files: eight implementations (six searchi files and two pinned llama.cpp files),
|
|
four technical documents, and twenty licensed media files. It is a small
|
|
single-annotator diagnostic, with reused previously evaluated media labels;
|
|
**these results do not establish production quality or untouched held-out quality**.
|
|
|
|
## Completed execution
|
|
|
|
The evaluator finished successfully with 210 rows: 104 executed requests per
|
|
mode plus one unsupported temporal-filter query per mode. All 32 files were
|
|
indexed; document extraction reported zero failures. Semantic indexing took
|
|
**1,060,840 ms (17 minutes 40.84 seconds)** and published generation one with
|
|
**182 chunks: 64 text and 118 media**, derived from 62 document passages and
|
|
20 media files. The final report records zero media failures and an empty
|
|
failure list. It made 62 document, 99 image/frame and 19 audio inference calls;
|
|
82 source tasks were encoded and none reused. Image/frame calls plus audio calls
|
|
account for the 118 media chunks. Indexing costs include decoder work and model
|
|
initialization; they are not pure native inference measurements.
|
|
|
|
| Positive queries | Hybrid All files top1 | Hybrid All files recall@5 |
|
|
| --- | --- | --- |
|
|
| Implementation intent | 10/14 | 13/14 |
|
|
| Technical documents | 7/8 | 8/8 |
|
|
| Images | 24/24 | 24/24 |
|
|
| Videos, file identity only | 22/22 | 22/22 |
|
|
| Music | 2/4 | 4/4 |
|
|
| Sounds | 4/4 | 4/4 |
|
|
| Exact media metadata/filter | 4/4 | 4/4 |
|
|
| **All positive queries** | **73/80** | **79/80** |
|
|
|
|
Lexical-only All files found the expected file first for **3/80** positives,
|
|
with **3/80** recall@5. In explicit Code scope, hybrid found **14/14** code
|
|
queries first; lexical-only found **0/14**. This covers two upstream files and
|
|
six self-repository implementations, not fourteen independent projects.
|
|
|
|
All **10/10 no-answer queries returned results** in hybrid All files;
|
|
lexical-only returned no results for those ten. This exposes uncalibrated
|
|
irrelevant neighbors, rather than proving an answer exists. Do not interpret
|
|
the displayed hybrid scores as confidence or tune a rejection threshold on
|
|
these same frozen labels. One query's requested temporal-range filter remains
|
|
unsupported and was explicitly skipped in each mode.
|
|
|
|
## Concrete failures
|
|
|
|
- English `code that chooses audio search when a query mentions speech` returned
|
|
audio matches and omitted the expected routing implementation entirely from
|
|
twenty results. Explicit Code scope placed that implementation first. The
|
|
German paired request found it first in both scopes. This confirms a real
|
|
All-files audio-intent suppression failure; the separate routing audit
|
|
explains the current rule. No production ranking change was made for this run.
|
|
- Both CPU-budget implementation queries ranked the performance documentation
|
|
above the implementation (rank two). The English rendering/queue timing
|
|
implementation query also placed the documentation first. Code scope resolved
|
|
these small cases, but requiring users to switch scope does not establish the
|
|
requested All-files experience.
|
|
- The German media-change limitations query ranked incremental-indexing
|
|
documentation above the intended automatic-indexing document (rank two).
|
|
Its desired source remained present in the first five.
|
|
- The English and German jazzy-violin queries ranked piano music first and
|
|
the correct violin music second. The frozen music hard negative remains a
|
|
genuine top-result failure.
|
|
|
|
Video/speech rows above measure **correct file identity**, not a useful scene or
|
|
spoken passage. They do not overturn the poor strict speech timestamp results
|
|
in [MEDIA-QUALITY.md](MEDIA-QUALITY.md). Returned source timestamps are retained
|
|
in the raw rows for further inspection. A correctly retrieved video can still
|
|
open at the wrong moment.
|
|
|
|
## Timing boundaries and evidence
|
|
|
|
Across 90 executed All-files requests in their fixed fixture order, observed
|
|
hybrid MCP roundtrip median/p95 were **101.98/136.38 ms**; native basic search
|
|
median/p95 were **94.22/125.90 ms**. Lexical roundtrip median/p95 were
|
|
**2.33/3.24 ms**, with native **1.31/2.14 ms**. The first hybrid request took
|
|
117.73 ms roundtrip and 107.04 ms natively. These are ordered tiny-corpus
|
|
observations, excluding process initialization/model loading; they are neither
|
|
first user-input timing nor randomized production-scale warm p95. Code-scope
|
|
requests alternate with All-files code queries, so task/filter caches and prior
|
|
requests are part of this run's conditions.
|
|
|
|
The complete raw report remains in ignored `.searchi/qa/mixed-v1.json` (636,101
|
|
bytes), SHA-256 `c6619d4a66332f0378d42b9ec4f108349ce095aeb2dbcccd535de6095438b37a`.
|
|
It retains every ordered hit, query, expected label, source location, native
|
|
metric, index stdout/stderr and final semantic report. Runtime stderr retained
|
|
Google token-type warnings and the native runtime's experimental audio warning;
|
|
these were not hidden as failed assets or omitted from evidence.
|
|
|
|
| Provenance | SHA-256 |
|
|
| --- | --- |
|
|
| Frozen labels | `f83d27227582f614159ff153654d5bb5a47bf74998141e389557f3950ea0b924` |
|
|
| Actual debug CLI | `23afd9986d987bc45db9e08aa50f1070f3ad2229d275793c1f2a41ee0fcea545` |
|
|
| Actual normal worker | `c01cb3ec13cfca722ee768e8350e1399a8b9a4d14a24a35adc3473a5cfc82a73` |
|
|
| Pinned Google model manifest | `fad82f171f4976d8eeb89c11d6a79fc46d92afb73c10a34f140d5b56e377397f` |
|
|
|
|
The worker/model pair was SHA-verified before inference. The active fingerprint
|
|
records query context 512 and document context 2048. The owned evaluator session
|
|
finished with exit zero and cleaned its temporary corpus/index and worker; no
|
|
restart or substituted model was used. Reproduce with the
|
|
[evaluator instructions](../tools/mixed-eval/README.md).
|
|
|
|
## Open quality gates
|
|
|
|
- Validate the subsequent explicit-code routing fix on broader mixed queries
|
|
without hiding legitimate media results. A tiny real-model candidate-preservation
|
|
regression passed; see [routing evidence](RETRIEVAL-ROUTING.md). The frozen
|
|
baseline recorded here remains the pre-fix measurement.
|
|
- Evaluate general/code query task fusion on a new independently labelled set,
|
|
with bounded retrieval and measured cache/filter costs.
|
|
- Improve music discrimination and precise speech timestamps.
|
|
- Establish no-answer behavior on a separate calibration/held-out split while
|
|
preserving positive recall.
|
|
- Add unrelated repositories, customer documents, longer recordings, human
|
|
listening and a second blinded annotator before broad relevance claims.
|