Freeze independently reviewed relevance calibration corpus

docs/RELEVANCE-CALIBRATION.md

@@ -0,0 +1,48 @@
+# Relevance calibration preparation
+
+The existing frozen mixed diagnostic returned candidates for all ten requests
+without a relevant source. That observation identifies an abstention problem;
+it does not justify choosing a cosine cutoff on those same queries.
+
+The new [text/code diagnostic](../fixtures/relevance-v2/README.md) separates
+calibration and held-out topic families and repositories before inference.
+Its 14 files support 48 English/German queries: 32 positive requests and 16
+requests for absent artifacts or implementations. Both translations of an
+intent stay in the same split. A broadly related neighbour may be useful for
+an exploratory topic query, so these absence labels require concrete artifacts.
+
+The source inventory and blinded review have been completed. Independent AI
+review agreed with all expected file sets; the final labels are frozen in
+`fixture.json` with hashes and a UTC timestamp in `fixture.lock.json`.
+No retrieval run, threshold selection or ranking change has occurred. Agent
+agreement will not be described as human review.
+
+The first independent agent review detected Windows-1252 mojibake in the
+authored German requests and ambiguity in the wildcard-comparator request.
+Before inference or freezing, German request strings were restored to Unicode
+and the comparator request clarified to a lone `*` followed by another
+comparator, such as `*, >=1.0.0`. Numeric partial versions such as `1.*` are
+outside that rejection claim. The original blinded input and per-query changes
+are preserved in `fixtures/relevance-v2/review-history`; upstream source and
+licence bytes were unchanged. The corrected independent re-read agreed with
+all expected file sets. Before freezing, two further paired requests were made
+more precise: bounded pattern search requests its first match, and the museum
+request asks for operative loan terms rather than an executed agreement.
+Both the preceding input and report remain byte-exact in review history;
+independent re-reading confirmed those four wording changes.
+
+An absent label means the requested artifact or implementation is missing.
+It does not mean no related file exists or that related files must never be
+shown as suggestions. Any future abstention policy must distinguish that
+semantics explicitly when it reports results.
+
+After freezing, use calibration only to propose an explicit relevance policy;
+**Keep held-out queries unevaluated until that policy and its parameters are fixed.**
+report positive recall alongside absent-query behaviour. Preserve raw candidates,
+native and end-to-end timings, model/runtime hashes, task scope and any unsupported
+filters. Evaluate the held-out split once without selecting parameters from it.
+If labels need correction after model output, create a new version and explain
+why instead of altering this diagnostic silently.
+
+This small corpus does not close independent human relevance, media abstention,
+music discrimination, speech localisation or production-scale quality gates.