2026-10-13 07:06:39 +02:00
|
|
|
# Mixed All-files diagnostic v1
|
|
|
|
|
|
2026-10-13 08:36:31 +02:00
|
|
|
This is a frozen, small diagnostic, **not a production quality score**. The first genuine CPU mixed
|
|
|
|
|
run is recorded in [MIXED-QUALITY.md](../../docs/MIXED-QUALITY.md); its failures
|
|
|
|
|
remain visible and v1 labels remain unchanged.
|
2026-10-13 07:06:39 +02:00
|
|
|
|
|
|
|
|
The corpus contains 32 actual files: six searchi implementations, four searchi
|
|
|
|
|
technical documents, two complete pinned llama.cpp source files, and twenty
|
|
|
|
|
existing licensed media files. Source snapshots retain exact original bytes;
|
|
|
|
|
`fixture.json` records upstream paths, commits, SHA-256, sizes, and licenses.
|
|
|
|
|
The llama.cpp copies come from committed upstream objects, excluding local
|
|
|
|
|
runtime patches. Its MIT notice is in `LLAMA-LICENSE.txt`. Media files are
|
|
|
|
|
referenced from `../media/assets` rather than duplicated. Their source URLs,
|
|
|
|
|
licenses, independently observed labels and attribution remain in
|
|
|
|
|
[the media fixture](../media/README.md).
|
|
|
|
|
|
|
|
|
|
There are 91 frozen queries: English/German implementation and document pairs,
|
|
|
|
|
65 reused media queries, four additional absent-content queries and four pinned
|
|
|
|
|
upstream implementation queries. Code queries run in both All files and Code
|
|
|
|
|
scope. Documents about the implementation coexist with the code, providing
|
|
|
|
|
realistic competing content. Speech, music, images and videos compete with
|
|
|
|
|
both. The case asking for code that chooses speech search is intentional: it
|
|
|
|
|
can expose audio-intent routing that suppresses a relevant implementation.
|
|
|
|
|
|
|
|
|
|
Labels were written by one annotator before **mixed** inference. Existing media
|
|
|
|
|
results had already been observed in their separate evaluation, so this is not
|
|
|
|
|
an untouched held-out media benchmark. searchi sources are the application's
|
|
|
|
|
own repository; the two llama.cpp files add external code, but do not establish
|
|
|
|
|
quality across unrelated real projects. Never adjust v1 labels after seeing
|
|
|
|
|
mixed results. Corrections require a new fixture version, explicit reasoning,
|
|
|
|
|
and a new pre-inference lock. A second blinded human review remains open.
|
|
|
|
|
|
|
|
|
|
Run `python tools/mixed-eval/evaluate.py --verify-only` to verify labels and all
|
|
|
|
|
source/media bytes without invoking a model. See
|
|
|
|
|
[the evaluator instructions](../../tools/mixed-eval/README.md) for real runs.
|
