searchi/tools/relevance-eval/README.md

# Frozen relevance diagnostic

## New absence holdout

`--bin absence` is a separate runner for `fixtures/absence-v3`, with exact
fixture and evidence-policy hashes enforced. It does not change the historical
v2 runner or its held-out gate. It materializes the 36 frozen source files into
an owned persistent profile, hashes existing copies before reuse, runs actual
Google media/text indexing with CPU speech recognition and reports per-asset
extraction errors and active semantic channels. A failed extraction or absent
required channel must not be described as successful abstention.

Set `SEARCHI_FORCE_CPU=1`, `SEARCHI_EXTRACT_WORKER`, `SEARCHI_GEMMA_MODEL_DIR`,
`SEARCHI_GEMMA_WORKER`, `SEARCHI_TEST_FFMPEG`, `SEARCHI_TEST_FFPROBE`,
`SEARCHI_SPEECH_WORKER`, `SEARCHI_SPEECH_BASE` and `SEARCHI_SPEECH_VAD` to verified
local assets. Reserve the shared CPU slot before running:

```powershell
cargo test --manifest-path tools/relevance-eval/Cargo.toml --bin absence --locked
cargo build --release --manifest-path tools/relevance-eval/Cargo.toml --bin absence --locked
tools/relevance-eval/target/release/absence.exe C:/path/to/searchi C:/path/to/searchi/.searchi/qa/absence-v3-runtime C:/path/to/searchi/.searchi/qa/absence-v3-results.json
```

Existing evidence outputs are never overwritten. Raw routed cosine results,
current primary and related results, and a replay of the previously shipped
leading-cluster hybrid rule are retained. The replay uses the actual vectors
and keyword candidates from this run; it is not execution or timing of an old
binary. Its fixture queries contain no identifier-specific search; it does not
replay the service's identifier promotion. Current search runs through the
shared service, including strict code scope. Results are capped at 20, with an
80-candidate budget. Primary absence counts and primary/accessibly retained
positive recall are separate; related suggestions are not claimed to be exact
answers. English/German and category metrics are reported separately. Four
partial requests are excluded from strict positive/absence counts.

Calls run lexical, then raw semantic, then current hybrid. Hybrid reuses the
query-vector cache, so these timings do not measure cold first user input. Do
not tune the frozen policy on this holdout. See the fixture README for label
and corpus limitations.

## Historical v2 diagnostic

This internal runner uses searchi's shared Rust engine and genuine Google
EmbeddingGemma 2. By default it indexes all 14 frozen source files as the fixed
distractor library and evaluates only the 24 calibration queries. Held-out
execution requires `--held-out-policy` with the exact SHA-256 of the hypothesis
frozen in commit `e7245fe`, before held-out inference. Changed policy bytes are
rejected. The two splits are selected independently and never mixed in one run.

First verify all frozen labels, source bytes, licenses and split integrity:

```powershell
python fixtures/relevance-v2/verify.py
cargo test --manifest-path tools/relevance-eval/Cargo.toml --locked
cargo build --release --manifest-path tools/relevance-eval/Cargo.toml --locked
```

Only run when the shared CPU inference slot is free:

```powershell
$env:SEARCHI_FORCE_CPU = '1'
tools/relevance-eval/target/release/searchi-relevance-eval.exe fixtures/relevance-v2 C:/path/to/verified-models C:/path/to/searchi-gemma-worker.exe .searchi/qa/calibration-v2.json
```

For the one frozen held-out evaluation, append
`--held-out-policy tools/relevance-eval/policies/cosine-v1.json` and use a new
held-out report name. Raw lexical, semantic and production hybrid results are
retained. `scores.hypothesis_semantic` evaluates raw semantic candidates after
the frozen cutoff; it is not a newly executed filtered hybrid fusion. This
diagnostic does not deploy a threshold into the shared engine. Do not retune
this policy using the held-out outcomes.

Supply an existing output directory and a new output filename. Existing evidence
is never overwritten: the output is reserved with `create_new` before costly
work and subsequent phase/row checkpoints replace JSON atomically. The runner verifies fixture SHA-256, exact source bytes,
and both pinned Google assets before indexing. It records runner/worker/model
hashes, actual worker capabilities, indexing/startup timings, all lexical
candidates (including match kinds and source locations), raw semantic cosine
scores, and final hybrid results with detailed engine traces. Code intents run
both All files and Code scope; all lanes use the same strict scope filters.

The final 20-result hybrid request uses an 80-candidate budget. The diagnostic
therefore records 80 lexical/semantic candidates separately. Semantic candidates
are the production routed, per-file lane, not all passage vectors. Hybrid scores
are reciprocal-rank-fusion scores, **not confidence**. No arbitrary cosine cutoff
is applied, no vectors are simulated, and no quality acceptance claim follows
from a successful run.

Queries execute lexical → semantic → hybrid in a retained query-role session.
Hybrid consequently reuses the preceding semantic query vector. These timings
are ordered diagnostic timings, not cold first input, UI latency, independent
warm p95, or production performance evidence. Partial completed rows are saved
after each request; an interrupted/failed report remains `complete: false`.

This corpus is small: fictional customer documents, two real source files and
independent AI annotation. It does not cover media or production scale. Preserve
labels and original raw output. A later policy experiment must fix all routing,
abstention and parameter choices on calibration only before exposing held-out
queries, and must retain exact filename priority and source provenance.