43 lines
2.5 KiB
Markdown
43 lines
2.5 KiB
Markdown
|
# Text and code relevance diagnostic
|
||
|
|
|
||
|
|
This frozen fixture contains 14 actual files and 48 English/German requests. It is a
|
||
|
|
small, agent-authored diagnostic, not a human-labelled production quality gate.
|
||
|
|
No inference has been performed against these labels.
|
||
|
|
|
||
|
|
`inventory.json` records exact source bytes, upstream immutable commits and
|
||
|
|
complete licence files. Twelve documents describe fictional organisations and
|
||
|
|
procedures; no customer information is used. Two complete Rust files come from
|
||
|
|
unrelated upstream projects. Local `.gitattributes` prevents Git newline
|
||
|
|
conversion of source and licence files.
|
||
|
|
|
||
|
|
Calibration uses bicycle warranty, bakery labels, apartment maintenance and
|
||
|
|
aho-corasick. Held-out uses museum loans, irrigation, ferry operations and
|
||
|
|
semver. Topic families and upstream repositories are disjoint. Each bilingual
|
||
|
|
intent pair stays within one split. Positive requests include contrasting
|
||
|
|
nearby procedures; absent requests require a specific missing artifact or
|
||
|
|
implementation. A related topic alone cannot fulfil such a request.
|
||
|
|
|
||
|
|
The independent agent reviewer should read only `blind-review-input.json` and
|
||
|
|
its referenced source files, then return candidate file IDs, an empty set for
|
||
|
|
an absent artifact, or an explicit ambiguity. Do not read `author-labels.json`
|
||
|
|
until that review is saved. No vector, search ranking or threshold is available
|
||
|
|
to either annotator. Disagreements must be resolved with written reasons before
|
||
|
|
a final immutable label lock is created.
|
||
|
|
|
||
|
|
The independent AI review is now saved and agreed with all expected file sets.
|
||
|
|
`fixture.json` combines the reviewed queries and labels; `fixture.lock.json`
|
||
|
|
records their hashes and the UTC freeze time before any inference. Review history
|
||
|
|
preserves encoding repairs and wording clarifications before that freeze.
|
||
|
|
**Do not evaluate held-out queries until a policy or threshold has been fixed
|
||
|
|
using calibration only.** Do not adjust that policy from held-out results.
|
||
|
|
|
||
|
|
No media is included: this corpus cannot calibrate music, images, video or
|
||
|
|
spoken-passage abstention. Existing media benchmarks remain separate. A held-out
|
||
|
|
split made by the same author is weaker than independent human annotation and
|
||
|
|
larger real customer libraries.
|
||
|
|
|
||
|
|
Run `python fixtures/relevance-v2/verify.py` for source integrity and split
|
||
|
|
structure. That check proves consistency, not retrieval quality.
|
||
|
|
Run `python fixtures/relevance-v2/test_verify.py` for isolated corruption,
|
||
|
|
duplicate, split-leak and lock regressions, including verifier execution with
|
||
|
|
Python optimization enabled.
|