151 lines
8.6 KiB
Markdown
151 lines
8.6 KiB
Markdown
# Relevance calibration preparation
|
|
|
|
The existing frozen mixed diagnostic returned candidates for all ten requests
|
|
without a relevant source. That observation identifies an abstention problem;
|
|
it does not justify choosing a cosine cutoff on those same queries.
|
|
|
|
The new [text/code diagnostic](../fixtures/relevance-v2/README.md) separates
|
|
calibration and held-out topic families and repositories before inference.
|
|
Its 14 files support 48 English/German queries: 32 positive requests and 16
|
|
requests for absent artifacts or implementations. Both translations of an
|
|
intent stay in the same split. A broadly related neighbour may be useful for
|
|
an exploratory topic query, so these absence labels require concrete artifacts.
|
|
|
|
The source inventory and blinded review have been completed. Independent AI
|
|
review agreed with all expected file sets; the final labels are frozen in
|
|
`fixture.json` with hashes and a UTC timestamp in `fixture.lock.json`.
|
|
The calibration-only baseline has run with genuine Google CPU inference.
|
|
No threshold selection or ranking change has occurred. Agent agreement will
|
|
not be described as human review.
|
|
|
|
The first independent agent review detected Windows-1252 mojibake in the
|
|
authored German requests and ambiguity in the wildcard-comparator request.
|
|
Before inference or freezing, German request strings were restored to Unicode
|
|
and the comparator request clarified to a lone `*` followed by another
|
|
comparator, such as `*, >=1.0.0`. Numeric partial versions such as `1.*` are
|
|
outside that rejection claim. The original blinded input and per-query changes
|
|
are preserved in `fixtures/relevance-v2/review-history`; upstream source and
|
|
licence bytes were unchanged. The corrected independent re-read agreed with
|
|
all expected file sets. Before freezing, two further paired requests were made
|
|
more precise: bounded pattern search requests its first match, and the museum
|
|
request asks for operative loan terms rather than an executed agreement.
|
|
Both the preceding input and report remain byte-exact in review history;
|
|
independent re-reading confirmed those four wording changes.
|
|
|
|
An absent label means the requested artifact or implementation is missing.
|
|
It does not mean no related file exists or that related files must never be
|
|
shown as suggestions. Any future abstention policy must distinguish that
|
|
semantics explicitly when it reports results.
|
|
|
|
After freezing, use calibration only to propose an explicit relevance policy;
|
|
**Keep held-out queries unevaluated until that policy and its parameters are fixed.**
|
|
report positive recall alongside absent-query behaviour. Preserve raw candidates,
|
|
native and end-to-end timings, model/runtime hashes, task scope and any unsupported
|
|
filters. Evaluate the held-out split once without selecting parameters from it.
|
|
If labels need correction after model output, create a new version and explain
|
|
why instead of altering this diagnostic silently.
|
|
|
|
This small corpus does not close independent human relevance, media abstention,
|
|
music discrimination, speech localisation or production-scale quality gates.
|
|
|
|
## Recorded CPU baseline, 2026-10-13
|
|
|
|
The shared-engine runner completed all 24 calibration requests plus six code
|
|
intent requests repeated under Code scope. Held-out requests were not evaluated.
|
|
All 14 frozen sources remained available as distractors.
|
|
|
|
| Scope | Positive requests | Lexical top-1 | Semantic / hybrid top-1 | Absent requests returning semantic / hybrid neighbours |
|
|
| --- | --- | --- | --- | --- |
|
|
| All files | 16 | 1/16 | 16/16 | 8/8 |
|
|
| Additional Code scope requests | 4 | 0/4 | 4/4 | 2/2 |
|
|
|
|
The additional code rows repeat intents; they are not independent observations.
|
|
Lexical search returned no results for the absent requests. Semantic similarity
|
|
still retrieves related neighbours when the requested source is absent. This
|
|
baseline therefore supports useful positive retrieval, but does not establish
|
|
reliable abstention or justify a cutoff chosen without further calibration.
|
|
|
|
Ignored raw evidence: `.searchi/qa/calibration-v2-baseline.json`, SHA-256
|
|
`56f42c523527dd748918bb8f17be6bd7818158227e692fccaf387d915f9d0a94`.
|
|
The report records completion, calibration split and `held_out_evaluated: false`.
|
|
Frozen fixture SHA-256:
|
|
`000047fd17accd315b6138093bbdc9f353cd877b4bf1958fbe336e4a6b10cd2e`.
|
|
|
|
This ordered debug diagnostic used one CPU thread and the normal worker
|
|
`c01cb3ec13cfca722ee768e8350e1399a8b9a4d14a24a35adc3473a5cfc82a73`.
|
|
Lexical indexing took 19.9823 ms, semantic indexing 74,545.518 ms and retained
|
|
query-session startup 7,433.3914 ms. These are diagnostic setup measurements,
|
|
not native first-search latency or warm production p95.
|
|
|
|
## Frozen diagnostic hypothesis, 2026-10-13
|
|
|
|
Before evaluating held-out requests, freeze
|
|
[`document-code-cosine-070-v1`](../tools/relevance-eval/policies/cosine-v1.json):
|
|
retain document/code semantic candidates with raw cosine **>= 0.70** before
|
|
hybrid fusion. Preserve lexical results and exact identifier priority unchanged.
|
|
This is a diagnostic hypothesis, not a deployed product change; media is excluded.
|
|
It is not a probability or a universal EmbeddingGemma threshold.
|
|
|
|
The calibration analyzer accepts only the exact previously recorded report SHA,
|
|
rejects held-out inputs and nonfinite scores, and uses inclusive comparison.
|
|
Three structural tests passed in normal and optimized Python modes.
|
|
|
|
| Scope | Minimum positive top cosine | Maximum absent top cosine | Retained positives / absent queries with retained neighbours |
|
|
| --- | --- | --- | --- |
|
|
| All files | 0.703227 | 0.693980 | 16/16 / 0/8 |
|
|
| Additional Code scope | 0.761020 | 0.656212 | 4/4 / 0/2 |
|
|
|
|
The All files separation is only **0.009247**. The rounded 0.70 hypothesis
|
|
fits this small calibration sample; this narrow separation is a reason to
|
|
expect generalization failures, not to claim reliable abstention. Bilingual
|
|
pairs and repeated Code scope requests are correlated observations.
|
|
No margin, language-specific thresholds, task-specific tuning or model changes
|
|
are added to this version. Evaluate held-out once without adjusting this policy
|
|
from its outcomes; a failure requires recording the failure, not retuning it
|
|
against the same held-out sample.
|
|
|
|
Policy SHA-256:
|
|
`df0a887b97dd7fda6aeff1b6c966fe7558521f40dbe1c911e91cfd312dca7461`.
|
|
Ignored analysis: `.searchi/qa/calibration-cosine-policy-v1.json`.
|
|
Held-out remains unevaluated at this freeze. Product ranking is unchanged.
|
|
|
|
## Held-out result and disposition, 2026-10-13
|
|
|
|
After the policy freeze was committed and pushed, actual Google CPU inference
|
|
completed all 24 held-out queries plus six Code-scope repeats. The fixture,
|
|
model, policy and threshold were unchanged. The explicit runner option requires
|
|
the exact pre-evaluation policy SHA; default execution remains calibration-only.
|
|
Four structural Rust tests and strict all-target Clippy passed.
|
|
|
|
| Scope | Unfiltered semantic / hybrid positive top-1 | Positive top results retained at 0.70 | Absent requests with retained semantic neighbours |
|
|
| --- | --- | --- | --- |
|
|
| All files | 16/16 | 15/16 | 0/8 |
|
|
| Additional Code scope | 4/4 | 4/4 | 0/2 |
|
|
|
|
The lost request is `version-wildcard-de`: its correct All files semantic
|
|
candidate scores **0.6766**, below the frozen threshold. It has no lexical
|
|
results, so preserving lexical retrieval would not rescue it. The same request
|
|
under Code scope scores **0.7388** and survives. The highest absent All files
|
|
top score is approximately **0.6954**, exceeding that positive score: there is
|
|
no single raw-cosine threshold separating these held-out top results.
|
|
|
|
**Reject deployment as a universal default cutoff.** Do not lower this policy
|
|
after seeing held-out outcomes or claim that its strong calibration result
|
|
proved safe abstention. Product ranking remains unchanged. The next relevance
|
|
work should preserve access to weak-but-correct results and evaluate richer
|
|
evidence on a new independent corpus; this sample is no longer unseen data.
|
|
|
|
`scores.hypothesis_semantic` measures retained semantic candidates only.
|
|
The recorded production hybrid request is unmodified; no filtered hybrid
|
|
fusion or UI confidence behavior was executed. This small bilingual corpus
|
|
does not establish general customer quality or media abstention.
|
|
|
|
Raw ignored report: `.searchi/qa/held-out-cosine-v1.json`, SHA-256
|
|
`8f6baceac27e90c719c2bbb522eeb2d1623056e2f8334f31f7dd67a912c0fccd`.
|
|
Diagnostic runner SHA-256:
|
|
`db685085f7f53d02cc85ef7c534e8ac02f4c5c806940bd4ff44ccbeae76e5762`.
|
|
Worker capabilities record Google EmbeddingGemma 2, CPU, one thread, query
|
|
role, context 512 and 768 dimensions. Semantic indexing took 73,991.6605 ms;
|
|
query startup took 7,153.4587 ms. These ordered debug setup measurements are
|
|
not UI latency or warm p95. Owned workers exited; the installed app was retained.
|