2.9 KiB
Relevance calibration preparation
The existing frozen mixed diagnostic returned candidates for all ten requests without a relevant source. That observation identifies an abstention problem; it does not justify choosing a cosine cutoff on those same queries.
The new text/code diagnostic separates calibration and held-out topic families and repositories before inference. Its 14 files support 48 English/German queries: 32 positive requests and 16 requests for absent artifacts or implementations. Both translations of an intent stay in the same split. A broadly related neighbour may be useful for an exploratory topic query, so these absence labels require concrete artifacts.
The source inventory and blinded review have been completed. Independent AI
review agreed with all expected file sets; the final labels are frozen in
fixture.json with hashes and a UTC timestamp in fixture.lock.json.
No retrieval run, threshold selection or ranking change has occurred. Agent
agreement will not be described as human review.
The first independent agent review detected Windows-1252 mojibake in the
authored German requests and ambiguity in the wildcard-comparator request.
Before inference or freezing, German request strings were restored to Unicode
and the comparator request clarified to a lone * followed by another
comparator, such as *, >=1.0.0. Numeric partial versions such as 1.* are
outside that rejection claim. The original blinded input and per-query changes
are preserved in fixtures/relevance-v2/review-history; upstream source and
licence bytes were unchanged. The corrected independent re-read agreed with
all expected file sets. Before freezing, two further paired requests were made
more precise: bounded pattern search requests its first match, and the museum
request asks for operative loan terms rather than an executed agreement.
Both the preceding input and report remain byte-exact in review history;
independent re-reading confirmed those four wording changes.
An absent label means the requested artifact or implementation is missing. It does not mean no related file exists or that related files must never be shown as suggestions. Any future abstention policy must distinguish that semantics explicitly when it reports results.
After freezing, use calibration only to propose an explicit relevance policy; Keep held-out queries unevaluated until that policy and its parameters are fixed. report positive recall alongside absent-query behaviour. Preserve raw candidates, native and end-to-end timings, model/runtime hashes, task scope and any unsupported filters. Evaluate the held-out split once without selecting parameters from it. If labels need correction after model output, create a new version and explain why instead of altering this diagnostic silently.
This small corpus does not close independent human relevance, media abstention, music discrimination, speech localisation or production-scale quality gates.