Freeze calibration-derived document and code relevance hypothesis
This commit is contained in:
1 parent
06eec103f6
commit
e7245fe15e
4 files changed
+163
No files matched your search
@@ -76,3 +76,35 @@ This ordered debug diagnostic used one CPU thread and the normal worker
|
||||
Lexical indexing took 19.9823 ms, semantic indexing 74,545.518 ms and retained
|
||||
query-session startup 7,433.3914 ms. These are diagnostic setup measurements,
|
||||
not native first-search latency or warm production p95.
|
||||
|
||||
## Frozen diagnostic hypothesis, 2026-10-13
|
||||
|
||||
Before evaluating held-out requests, freeze
|
||||
[`document-code-cosine-070-v1`](../tools/relevance-eval/policies/cosine-v1.json):
|
||||
retain document/code semantic candidates with raw cosine **>= 0.70** before
|
||||
hybrid fusion. Preserve lexical results and exact identifier priority unchanged.
|
||||
This is a diagnostic hypothesis, not a deployed product change; media is excluded.
|
||||
It is not a probability or a universal EmbeddingGemma threshold.
|
||||
|
||||
The calibration analyzer accepts only the exact previously recorded report SHA,
|
||||
rejects held-out inputs and nonfinite scores, and uses inclusive comparison.
|
||||
Three structural tests passed in normal and optimized Python modes.
|
||||
|
||||
| Scope | Minimum positive top cosine | Maximum absent top cosine | Retained positives / absent queries with retained neighbours |
|
||||
| --- | --- | --- | --- |
|
||||
| All files | 0.703227 | 0.693980 | 16/16 / 0/8 |
|
||||
| Additional Code scope | 0.761020 | 0.656212 | 4/4 / 0/2 |
|
||||
|
||||
The All files separation is only **0.009247**. The rounded 0.70 hypothesis
|
||||
fits this small calibration sample; this narrow separation is a reason to
|
||||
expect generalization failures, not to claim reliable abstention. Bilingual
|
||||
pairs and repeated Code scope requests are correlated observations.
|
||||
No margin, language-specific thresholds, task-specific tuning or model changes
|
||||
are added to this version. Evaluate held-out once without adjusting this policy
|
||||
from its outcomes; a failure requires recording the failure, not retuning it
|
||||
against the same held-out sample.
|
||||
|
||||
Policy SHA-256:
|
||||
`df0a887b97dd7fda6aeff1b6c966fe7558521f40dbe1c911e91cfd312dca7461`.
|
||||
Ignored analysis: `.searchi/qa/calibration-cosine-policy-v1.json`.
|
||||
Held-out remains unevaluated at this freeze. Product ranking is unchanged.
|
||||
Reference in new issue
Block a user