1.9 KiB
Mixed retrieval quality gate
A frozen mixed diagnostic is prepared in fixtures/mixed. It addresses the missing comparison between natural-language All-files queries and explicit Code scope while images, music, speech, videos and actual documents compete. It contains 32 real files and 91 English/German queries, including two pinned upstream llama.cpp implementations. The existing media labels are reused with their licenses and attribution. They are not an untouched held-out set.
No actual mixed inference results are recorded yet. The frozen manifest
SHA-256 is f83d27227582f614159ff153654d5bb5a47bf74998141e389557f3950ea0b924.
Asset/hash verification and six meaningful scoring/provenance tests passed.
These prove harness behavior only; they establish no retrieval accuracy.
The evaluator calls the shared production engine through CLI indexing and MCP search. It compares lexical/hybrid retrieval and All-files/Code scope without changing ranking or tuning a cutoff against these labels. File-level recall and no-answer returned-result rates remain separate from spoken timestamp localization. Raw evidence must preserve actual failures, encoding errors and unsupported filters.
Remaining work:
- Run the frozen corpus with the verified Google model and CPU runtime in a coordinated slot; record exact CLI/worker hashes and actual raw results.
- Inspect failures by modality, language, requested artifact type and scope; distinguish correct-file retrieval from a useful passage/timestamp.
- Author a larger independently reviewed held-out corpus before ranking changes; frozen diagnostics must not become the sole optimization target.
- Establish recall and no-answer acceptance targets on representative user libraries, unrelated repositories and longer recordings. Tiny diagnostics cannot close the production relevance gate.