searchi/tools/relevance-eval/README.md

Frozen relevance diagnostic

New absence holdout

--bin absence is a separate runner for fixtures/absence-v3, with exact fixture and evidence-policy hashes enforced. It does not change the historical v2 runner or its held-out gate. It materializes the 36 frozen source files into an owned persistent profile, hashes existing copies before reuse, runs actual Google media/text indexing with CPU speech recognition and reports per-asset extraction errors and active semantic channels. A failed extraction or absent required channel must not be described as successful abstention.

Set SEARCHI_FORCE_CPU=1, SEARCHI_EXTRACT_WORKER, SEARCHI_GEMMA_MODEL_DIR, SEARCHI_GEMMA_WORKER, SEARCHI_TEST_FFMPEG, SEARCHI_TEST_FFPROBE, SEARCHI_SPEECH_WORKER, SEARCHI_SPEECH_BASE and SEARCHI_SPEECH_VAD to verified local assets. Reserve the shared CPU slot before running:

cargo test --manifest-path tools/relevance-eval/Cargo.toml --bin absence --locked
cargo build --release --manifest-path tools/relevance-eval/Cargo.toml --bin absence --locked
tools/relevance-eval/target/release/absence.exe C:/path/to/searchi C:/path/to/searchi/.searchi/qa/absence-v3-runtime C:/path/to/searchi/.searchi/qa/absence-v3-results.json

Existing evidence outputs are never overwritten. Raw routed cosine results, current primary and related results, and a replay of the previously shipped leading-cluster hybrid rule are retained. The replay uses the actual vectors and keyword candidates from this run; it is not execution or timing of an old binary. Its fixture queries contain no identifier-specific search; it does not replay the service's identifier promotion. Current search runs through the shared service, including strict code scope. Results are capped at 20, with an 80-candidate budget. Primary absence counts and primary/accessibly retained positive recall are separate; related suggestions are not claimed to be exact answers. English/German and category metrics are reported separately. Four partial requests are excluded from strict positive/absence counts.

Calls run lexical, then raw semantic, then current hybrid. Hybrid reuses the query-vector cache, so these timings do not measure cold first user input. Do not tune the frozen policy on this holdout. See the fixture README for label and corpus limitations.

Historical v2 diagnostic

This internal runner uses searchi's shared Rust engine and genuine Google EmbeddingGemma 2. By default it indexes all 14 frozen source files as the fixed distractor library and evaluates only the 24 calibration queries. Held-out execution requires --held-out-policy with the exact SHA-256 of the hypothesis frozen in commit e7245fe, before held-out inference. Changed policy bytes are rejected. The two splits are selected independently and never mixed in one run.

First verify all frozen labels, source bytes, licenses and split integrity:

python fixtures/relevance-v2/verify.py
cargo test --manifest-path tools/relevance-eval/Cargo.toml --locked
cargo build --release --manifest-path tools/relevance-eval/Cargo.toml --locked

Only run when the shared CPU inference slot is free:

$env:SEARCHI_FORCE_CPU = '1'
tools/relevance-eval/target/release/searchi-relevance-eval.exe fixtures/relevance-v2 C:/path/to/verified-models C:/path/to/searchi-gemma-worker.exe .searchi/qa/calibration-v2.json

For the one frozen held-out evaluation, append --held-out-policy tools/relevance-eval/policies/cosine-v1.json and use a new held-out report name. Raw lexical, semantic and production hybrid results are retained. scores.hypothesis_semantic evaluates raw semantic candidates after the frozen cutoff; it is not a newly executed filtered hybrid fusion. This diagnostic does not deploy a threshold into the shared engine. Do not retune this policy using the held-out outcomes.

Supply an existing output directory and a new output filename. Existing evidence is never overwritten: the output is reserved with create_new before costly work and subsequent phase/row checkpoints replace JSON atomically. The runner verifies fixture SHA-256, exact source bytes, and both pinned Google assets before indexing. It records runner/worker/model hashes, actual worker capabilities, indexing/startup timings, all lexical candidates (including match kinds and source locations), raw semantic cosine scores, and final hybrid results with detailed engine traces. Code intents run both All files and Code scope; all lanes use the same strict scope filters.

The final 20-result hybrid request uses an 80-candidate budget. The diagnostic therefore records 80 lexical/semantic candidates separately. Semantic candidates are the production routed, per-file lane, not all passage vectors. Hybrid scores are reciprocal-rank-fusion scores, not confidence. No arbitrary cosine cutoff is applied, no vectors are simulated, and no quality acceptance claim follows from a successful run.

Queries execute lexical → semantic → hybrid in a retained query-role session. Hybrid consequently reuses the preceding semantic query vector. These timings are ordered diagnostic timings, not cold first input, UI latency, independent warm p95, or production performance evidence. Partial completed rows are saved after each request; an interrupted/failed report remains complete: false.

This corpus is small: fictional customer documents, two real source files and independent AI annotation. It does not cover media or production scale. Preserve labels and original raw output. A later policy experiment must fix all routing, abstention and parameter choices on calibration only before exposing held-out queries, and must retain exact filename priority and source provenance.