111 lines
6.3 KiB
Markdown
111 lines
6.3 KiB
Markdown
# Relevance calibration preparation
|
|
|
|
The existing frozen mixed diagnostic returned candidates for all ten requests
|
|
without a relevant source. That observation identifies an abstention problem;
|
|
it does not justify choosing a cosine cutoff on those same queries.
|
|
|
|
The new [text/code diagnostic](../fixtures/relevance-v2/README.md) separates
|
|
calibration and held-out topic families and repositories before inference.
|
|
Its 14 files support 48 English/German queries: 32 positive requests and 16
|
|
requests for absent artifacts or implementations. Both translations of an
|
|
intent stay in the same split. A broadly related neighbour may be useful for
|
|
an exploratory topic query, so these absence labels require concrete artifacts.
|
|
|
|
The source inventory and blinded review have been completed. Independent AI
|
|
review agreed with all expected file sets; the final labels are frozen in
|
|
`fixture.json` with hashes and a UTC timestamp in `fixture.lock.json`.
|
|
The calibration-only baseline has run with genuine Google CPU inference.
|
|
No threshold selection or ranking change has occurred. Agent agreement will
|
|
not be described as human review.
|
|
|
|
The first independent agent review detected Windows-1252 mojibake in the
|
|
authored German requests and ambiguity in the wildcard-comparator request.
|
|
Before inference or freezing, German request strings were restored to Unicode
|
|
and the comparator request clarified to a lone `*` followed by another
|
|
comparator, such as `*, >=1.0.0`. Numeric partial versions such as `1.*` are
|
|
outside that rejection claim. The original blinded input and per-query changes
|
|
are preserved in `fixtures/relevance-v2/review-history`; upstream source and
|
|
licence bytes were unchanged. The corrected independent re-read agreed with
|
|
all expected file sets. Before freezing, two further paired requests were made
|
|
more precise: bounded pattern search requests its first match, and the museum
|
|
request asks for operative loan terms rather than an executed agreement.
|
|
Both the preceding input and report remain byte-exact in review history;
|
|
independent re-reading confirmed those four wording changes.
|
|
|
|
An absent label means the requested artifact or implementation is missing.
|
|
It does not mean no related file exists or that related files must never be
|
|
shown as suggestions. Any future abstention policy must distinguish that
|
|
semantics explicitly when it reports results.
|
|
|
|
After freezing, use calibration only to propose an explicit relevance policy;
|
|
**Keep held-out queries unevaluated until that policy and its parameters are fixed.**
|
|
report positive recall alongside absent-query behaviour. Preserve raw candidates,
|
|
native and end-to-end timings, model/runtime hashes, task scope and any unsupported
|
|
filters. Evaluate the held-out split once without selecting parameters from it.
|
|
If labels need correction after model output, create a new version and explain
|
|
why instead of altering this diagnostic silently.
|
|
|
|
This small corpus does not close independent human relevance, media abstention,
|
|
music discrimination, speech localisation or production-scale quality gates.
|
|
|
|
## Recorded CPU baseline, 2026-10-13
|
|
|
|
The shared-engine runner completed all 24 calibration requests plus six code
|
|
intent requests repeated under Code scope. Held-out requests were not evaluated.
|
|
All 14 frozen sources remained available as distractors.
|
|
|
|
| Scope | Positive requests | Lexical top-1 | Semantic / hybrid top-1 | Absent requests returning semantic / hybrid neighbours |
|
|
| --- | --- | --- | --- | --- |
|
|
| All files | 16 | 1/16 | 16/16 | 8/8 |
|
|
| Additional Code scope requests | 4 | 0/4 | 4/4 | 2/2 |
|
|
|
|
The additional code rows repeat intents; they are not independent observations.
|
|
Lexical search returned no results for the absent requests. Semantic similarity
|
|
still retrieves related neighbours when the requested source is absent. This
|
|
baseline therefore supports useful positive retrieval, but does not establish
|
|
reliable abstention or justify a cutoff chosen without further calibration.
|
|
|
|
Ignored raw evidence: `.searchi/qa/calibration-v2-baseline.json`, SHA-256
|
|
`56f42c523527dd748918bb8f17be6bd7818158227e692fccaf387d915f9d0a94`.
|
|
The report records completion, calibration split and `held_out_evaluated: false`.
|
|
Frozen fixture SHA-256:
|
|
`000047fd17accd315b6138093bbdc9f353cd877b4bf1958fbe336e4a6b10cd2e`.
|
|
|
|
This ordered debug diagnostic used one CPU thread and the normal worker
|
|
`c01cb3ec13cfca722ee768e8350e1399a8b9a4d14a24a35adc3473a5cfc82a73`.
|
|
Lexical indexing took 19.9823 ms, semantic indexing 74,545.518 ms and retained
|
|
query-session startup 7,433.3914 ms. These are diagnostic setup measurements,
|
|
not native first-search latency or warm production p95.
|
|
|
|
## Frozen diagnostic hypothesis, 2026-10-13
|
|
|
|
Before evaluating held-out requests, freeze
|
|
[`document-code-cosine-070-v1`](../tools/relevance-eval/policies/cosine-v1.json):
|
|
retain document/code semantic candidates with raw cosine **>= 0.70** before
|
|
hybrid fusion. Preserve lexical results and exact identifier priority unchanged.
|
|
This is a diagnostic hypothesis, not a deployed product change; media is excluded.
|
|
It is not a probability or a universal EmbeddingGemma threshold.
|
|
|
|
The calibration analyzer accepts only the exact previously recorded report SHA,
|
|
rejects held-out inputs and nonfinite scores, and uses inclusive comparison.
|
|
Three structural tests passed in normal and optimized Python modes.
|
|
|
|
| Scope | Minimum positive top cosine | Maximum absent top cosine | Retained positives / absent queries with retained neighbours |
|
|
| --- | --- | --- | --- |
|
|
| All files | 0.703227 | 0.693980 | 16/16 / 0/8 |
|
|
| Additional Code scope | 0.761020 | 0.656212 | 4/4 / 0/2 |
|
|
|
|
The All files separation is only **0.009247**. The rounded 0.70 hypothesis
|
|
fits this small calibration sample; this narrow separation is a reason to
|
|
expect generalization failures, not to claim reliable abstention. Bilingual
|
|
pairs and repeated Code scope requests are correlated observations.
|
|
No margin, language-specific thresholds, task-specific tuning or model changes
|
|
are added to this version. Evaluate held-out once without adjusting this policy
|
|
from its outcomes; a failure requires recording the failure, not retuning it
|
|
against the same held-out sample.
|
|
|
|
Policy SHA-256:
|
|
`df0a887b97dd7fda6aeff1b6c966fe7558521f40dbe1c911e91cfd312dca7461`.
|
|
Ignored analysis: `.searchi/qa/calibration-cosine-policy-v1.json`.
|
|
Held-out remains unevaluated at this freeze. Product ranking is unchanged.
|