49 lines
2.9 KiB
Markdown
49 lines
2.9 KiB
Markdown
# Relevance calibration preparation
|
|
|
|
The existing frozen mixed diagnostic returned candidates for all ten requests
|
|
without a relevant source. That observation identifies an abstention problem;
|
|
it does not justify choosing a cosine cutoff on those same queries.
|
|
|
|
The new [text/code diagnostic](../fixtures/relevance-v2/README.md) separates
|
|
calibration and held-out topic families and repositories before inference.
|
|
Its 14 files support 48 English/German queries: 32 positive requests and 16
|
|
requests for absent artifacts or implementations. Both translations of an
|
|
intent stay in the same split. A broadly related neighbour may be useful for
|
|
an exploratory topic query, so these absence labels require concrete artifacts.
|
|
|
|
The source inventory and blinded review have been completed. Independent AI
|
|
review agreed with all expected file sets; the final labels are frozen in
|
|
`fixture.json` with hashes and a UTC timestamp in `fixture.lock.json`.
|
|
No retrieval run, threshold selection or ranking change has occurred. Agent
|
|
agreement will not be described as human review.
|
|
|
|
The first independent agent review detected Windows-1252 mojibake in the
|
|
authored German requests and ambiguity in the wildcard-comparator request.
|
|
Before inference or freezing, German request strings were restored to Unicode
|
|
and the comparator request clarified to a lone `*` followed by another
|
|
comparator, such as `*, >=1.0.0`. Numeric partial versions such as `1.*` are
|
|
outside that rejection claim. The original blinded input and per-query changes
|
|
are preserved in `fixtures/relevance-v2/review-history`; upstream source and
|
|
licence bytes were unchanged. The corrected independent re-read agreed with
|
|
all expected file sets. Before freezing, two further paired requests were made
|
|
more precise: bounded pattern search requests its first match, and the museum
|
|
request asks for operative loan terms rather than an executed agreement.
|
|
Both the preceding input and report remain byte-exact in review history;
|
|
independent re-reading confirmed those four wording changes.
|
|
|
|
An absent label means the requested artifact or implementation is missing.
|
|
It does not mean no related file exists or that related files must never be
|
|
shown as suggestions. Any future abstention policy must distinguish that
|
|
semantics explicitly when it reports results.
|
|
|
|
After freezing, use calibration only to propose an explicit relevance policy;
|
|
**Keep held-out queries unevaluated until that policy and its parameters are fixed.**
|
|
report positive recall alongside absent-query behaviour. Preserve raw candidates,
|
|
native and end-to-end timings, model/runtime hashes, task scope and any unsupported
|
|
filters. Evaluate the held-out split once without selecting parameters from it.
|
|
If labels need correction after model output, create a new version and explain
|
|
why instead of altering this diagnostic silently.
|
|
|
|
This small corpus does not close independent human relevance, media abstention,
|
|
music discrimination, speech localisation or production-scale quality gates.
|