Freeze calibration-derived document and code relevance hypothesis

This commit is contained in:
karl committed 2026-10-13 12:17:37 +02:00
1 parent 06eec103f6
commit e7245fe15e
4 files changed
+163

No files matched your search

+32
View File
@@ -76,3 +76,35 @@ This ordered debug diagnostic used one CPU thread and the normal worker
Lexical indexing took 19.9823 ms, semantic indexing 74,545.518 ms and retained
query-session startup 7,433.3914 ms. These are diagnostic setup measurements,
not native first-search latency or warm production p95.
## Frozen diagnostic hypothesis, 2026-10-13
Before evaluating held-out requests, freeze
[`document-code-cosine-070-v1`](../tools/relevance-eval/policies/cosine-v1.json):
retain document/code semantic candidates with raw cosine **>= 0.70** before
hybrid fusion. Preserve lexical results and exact identifier priority unchanged.
This is a diagnostic hypothesis, not a deployed product change; media is excluded.
It is not a probability or a universal EmbeddingGemma threshold.
The calibration analyzer accepts only the exact previously recorded report SHA,
rejects held-out inputs and nonfinite scores, and uses inclusive comparison.
Three structural tests passed in normal and optimized Python modes.
| Scope | Minimum positive top cosine | Maximum absent top cosine | Retained positives / absent queries with retained neighbours |
| --- | --- | --- | --- |
| All files | 0.703227 | 0.693980 | 16/16 / 0/8 |
| Additional Code scope | 0.761020 | 0.656212 | 4/4 / 0/2 |
The All files separation is only **0.009247**. The rounded 0.70 hypothesis
fits this small calibration sample; this narrow separation is a reason to
expect generalization failures, not to claim reliable abstention. Bilingual
pairs and repeated Code scope requests are correlated observations.
No margin, language-specific thresholds, task-specific tuning or model changes
are added to this version. Evaluate held-out once without adjusting this policy
from its outcomes; a failure requires recording the failure, not retuning it
against the same held-out sample.
Policy SHA-256:
`df0a887b97dd7fda6aeff1b6c966fe7558521f40dbe1c911e91cfd312dca7461`.
Ignored analysis: `.searchi/qa/calibration-cosine-policy-v1.json`.
Held-out remains unevaluated at this freeze. Product ranking is unchanged.