Distinguish main matches from related suggestions and ship 0.1.9
This commit is contained in:
1 parent
a8ba594d0e
commit
b0bdfc7816
40 files changed
+2463
-185
No files matched your search
@@ -1,5 +1,45 @@
|
||||
# Frozen relevance diagnostic
|
||||
|
||||
## New absence holdout
|
||||
|
||||
`--bin absence` is a separate runner for `fixtures/absence-v3`, with exact
|
||||
fixture and evidence-policy hashes enforced. It does not change the historical
|
||||
v2 runner or its held-out gate. It materializes the 36 frozen source files into
|
||||
an owned persistent profile, hashes existing copies before reuse, runs actual
|
||||
Google media/text indexing with CPU speech recognition and reports per-asset
|
||||
extraction errors and active semantic channels. A failed extraction or absent
|
||||
required channel must not be described as successful abstention.
|
||||
|
||||
Set `SEARCHI_FORCE_CPU=1`, `SEARCHI_EXTRACT_WORKER`, `SEARCHI_GEMMA_MODEL_DIR`,
|
||||
`SEARCHI_GEMMA_WORKER`, `SEARCHI_TEST_FFMPEG`, `SEARCHI_TEST_FFPROBE`,
|
||||
`SEARCHI_SPEECH_WORKER`, `SEARCHI_SPEECH_BASE` and `SEARCHI_SPEECH_VAD` to verified
|
||||
local assets. Reserve the shared CPU slot before running:
|
||||
|
||||
```powershell
|
||||
cargo test --manifest-path tools/relevance-eval/Cargo.toml --bin absence --locked
|
||||
cargo build --release --manifest-path tools/relevance-eval/Cargo.toml --bin absence --locked
|
||||
tools/relevance-eval/target/release/absence.exe C:/path/to/searchi C:/path/to/searchi/.searchi/qa/absence-v3-runtime C:/path/to/searchi/.searchi/qa/absence-v3-results.json
|
||||
```
|
||||
|
||||
Existing evidence outputs are never overwritten. Raw routed cosine results,
|
||||
current primary and related results, and a replay of the previously shipped
|
||||
leading-cluster hybrid rule are retained. The replay uses the actual vectors
|
||||
and keyword candidates from this run; it is not execution or timing of an old
|
||||
binary. Its fixture queries contain no identifier-specific search; it does not
|
||||
replay the service's identifier promotion. Current search runs through the
|
||||
shared service, including strict code scope. Results are capped at 20, with an
|
||||
80-candidate budget. Primary absence counts and primary/accessibly retained
|
||||
positive recall are separate; related suggestions are not claimed to be exact
|
||||
answers. English/German and category metrics are reported separately. Four
|
||||
partial requests are excluded from strict positive/absence counts.
|
||||
|
||||
Calls run lexical, then raw semantic, then current hybrid. Hybrid reuses the
|
||||
query-vector cache, so these timings do not measure cold first user input. Do
|
||||
not tune the frozen policy on this holdout. See the fixture README for label
|
||||
and corpus limitations.
|
||||
|
||||
## Historical v2 diagnostic
|
||||
|
||||
This internal runner uses searchi's shared Rust engine and genuine Google
|
||||
EmbeddingGemma 2. By default it indexes all 14 frozen source files as the fixed
|
||||
distractor library and evaluates only the 24 calibration queries. Held-out
|
||||
|
||||
Reference in new issue
Block a user