32 lines
1.9 KiB
Markdown
32 lines
1.9 KiB
Markdown
|
# Mixed retrieval quality gate
|
||
|
|
|
||
|
|
A frozen mixed diagnostic is prepared in [fixtures/mixed](../fixtures/mixed/README.md).
|
||
|
|
It addresses the missing comparison between natural-language All-files queries
|
||
|
|
and explicit Code scope while images, music, speech, videos and actual documents
|
||
|
|
compete. It contains 32 real files and 91 English/German queries, including two
|
||
|
|
pinned upstream llama.cpp implementations. The existing media labels are reused
|
||
|
|
with their licenses and attribution. They are not an untouched held-out set.
|
||
|
|
|
||
|
|
**No actual mixed inference results are recorded yet.** The frozen manifest
|
||
|
|
SHA-256 is `f83d27227582f614159ff153654d5bb5a47bf74998141e389557f3950ea0b924`.
|
||
|
|
Asset/hash verification and six meaningful scoring/provenance tests passed.
|
||
|
|
These prove harness behavior only; they establish no retrieval accuracy.
|
||
|
|
|
||
|
|
The [evaluator](../tools/mixed-eval/README.md) calls the shared production engine
|
||
|
|
through CLI indexing and MCP search. It compares lexical/hybrid retrieval and
|
||
|
|
All-files/Code scope without changing ranking or tuning a cutoff against these
|
||
|
|
labels. File-level recall and no-answer returned-result rates remain separate
|
||
|
|
from spoken timestamp localization. Raw evidence must preserve actual failures,
|
||
|
|
encoding errors and unsupported filters.
|
||
|
|
|
||
|
|
Remaining work:
|
||
|
|
|
||
|
|
- Run the frozen corpus with the verified Google model and CPU runtime in a
|
||
|
|
coordinated slot; record exact CLI/worker hashes and actual raw results.
|
||
|
|
- Inspect failures by modality, language, requested artifact type and scope;
|
||
|
|
distinguish correct-file retrieval from a useful passage/timestamp.
|
||
|
|
- Author a larger independently reviewed held-out corpus before ranking changes;
|
||
|
|
frozen diagnostics must not become the sole optimization target.
|
||
|
|
- Establish recall and no-answer acceptance targets on representative user
|
||
|
|
libraries, unrelated repositories and longer recordings. Tiny diagnostics
|
||
|
|
cannot close the production relevance gate.
|