Distinguish main matches from related suggestions and ship 0.1.9

This commit is contained in:
karl committed 2026-10-14 12:45:32 +02:00
1 parent a8ba594d0e
commit b0bdfc7816
40 files changed
+2463 -185

No files matched your search

+40
View File
@@ -1,5 +1,45 @@
# Frozen relevance diagnostic
## New absence holdout
`--bin absence` is a separate runner for `fixtures/absence-v3`, with exact
fixture and evidence-policy hashes enforced. It does not change the historical
v2 runner or its held-out gate. It materializes the 36 frozen source files into
an owned persistent profile, hashes existing copies before reuse, runs actual
Google media/text indexing with CPU speech recognition and reports per-asset
extraction errors and active semantic channels. A failed extraction or absent
required channel must not be described as successful abstention.
Set `SEARCHI_FORCE_CPU=1`, `SEARCHI_EXTRACT_WORKER`, `SEARCHI_GEMMA_MODEL_DIR`,
`SEARCHI_GEMMA_WORKER`, `SEARCHI_TEST_FFMPEG`, `SEARCHI_TEST_FFPROBE`,
`SEARCHI_SPEECH_WORKER`, `SEARCHI_SPEECH_BASE` and `SEARCHI_SPEECH_VAD` to verified
local assets. Reserve the shared CPU slot before running:
```powershell
cargo test --manifest-path tools/relevance-eval/Cargo.toml --bin absence --locked
cargo build --release --manifest-path tools/relevance-eval/Cargo.toml --bin absence --locked
tools/relevance-eval/target/release/absence.exe C:/path/to/searchi C:/path/to/searchi/.searchi/qa/absence-v3-runtime C:/path/to/searchi/.searchi/qa/absence-v3-results.json
```
Existing evidence outputs are never overwritten. Raw routed cosine results,
current primary and related results, and a replay of the previously shipped
leading-cluster hybrid rule are retained. The replay uses the actual vectors
and keyword candidates from this run; it is not execution or timing of an old
binary. Its fixture queries contain no identifier-specific search; it does not
replay the service's identifier promotion. Current search runs through the
shared service, including strict code scope. Results are capped at 20, with an
80-candidate budget. Primary absence counts and primary/accessibly retained
positive recall are separate; related suggestions are not claimed to be exact
answers. English/German and category metrics are reported separately. Four
partial requests are excluded from strict positive/absence counts.
Calls run lexical, then raw semantic, then current hybrid. Hybrid reuses the
query-vector cache, so these timings do not measure cold first user input. Do
not tune the frozen policy on this holdout. See the fixture README for label
and corpus limitations.
## Historical v2 diagnostic
This internal runner uses searchi's shared Rust engine and genuine Google
EmbeddingGemma 2. By default it indexes all 14 frozen source files as the fixed
distractor library and evaluates only the 24 calibration queries. Held-out