35 lines
2.0 KiB
Markdown
35 lines
2.0 KiB
Markdown
# Mixed All-files diagnostic v1
|
|
|
|
This is a frozen, small diagnostic, **not a production quality score**. No mixed
|
|
model run has been performed yet.
|
|
|
|
The corpus contains 32 actual files: six searchi implementations, four searchi
|
|
technical documents, two complete pinned llama.cpp source files, and twenty
|
|
existing licensed media files. Source snapshots retain exact original bytes;
|
|
`fixture.json` records upstream paths, commits, SHA-256, sizes, and licenses.
|
|
The llama.cpp copies come from committed upstream objects, excluding local
|
|
runtime patches. Its MIT notice is in `LLAMA-LICENSE.txt`. Media files are
|
|
referenced from `../media/assets` rather than duplicated. Their source URLs,
|
|
licenses, independently observed labels and attribution remain in
|
|
[the media fixture](../media/README.md).
|
|
|
|
There are 91 frozen queries: English/German implementation and document pairs,
|
|
65 reused media queries, four additional absent-content queries and four pinned
|
|
upstream implementation queries. Code queries run in both All files and Code
|
|
scope. Documents about the implementation coexist with the code, providing
|
|
realistic competing content. Speech, music, images and videos compete with
|
|
both. The case asking for code that chooses speech search is intentional: it
|
|
can expose audio-intent routing that suppresses a relevant implementation.
|
|
|
|
Labels were written by one annotator before **mixed** inference. Existing media
|
|
results had already been observed in their separate evaluation, so this is not
|
|
an untouched held-out media benchmark. searchi sources are the application's
|
|
own repository; the two llama.cpp files add external code, but do not establish
|
|
quality across unrelated real projects. Never adjust v1 labels after seeing
|
|
mixed results. Corrections require a new fixture version, explicit reasoning,
|
|
and a new pre-inference lock. A second blinded human review remains open.
|
|
|
|
Run `python tools/mixed-eval/evaluate.py --verify-only` to verify labels and all
|
|
source/media bytes without invoking a model. See
|
|
[the evaluator instructions](../../tools/mixed-eval/README.md) for real runs.
|