This internal runner uses searchi's shared Rust engine and genuine Google EmbeddingGemma 2. By default it indexes all 14 frozen source files as the fixed distractor library and evaluates only the 24 calibration queries. Held-out execution requires --held-out-policy with the exact SHA-256 of the hypothesis frozen in commit e7245fe, before held-out inference. Changed policy bytes are rejected. The two splits are selected independently and never mixed in one run.

First verify all frozen labels, source bytes, licenses and split integrity:

python fixtures/relevance-v2/verify.py
cargo test --manifest-path tools/relevance-eval/Cargo.toml --locked
cargo build --release --manifest-path tools/relevance-eval/Cargo.toml --locked

Only run when the shared CPU inference slot is free:

$env:SEARCHI_FORCE_CPU = '1'
tools/relevance-eval/target/release/searchi-relevance-eval.exe fixtures/relevance-v2 C:/path/to/verified-models C:/path/to/searchi-gemma-worker.exe .searchi/qa/calibration-v2.json

For the one frozen held-out evaluation, append --held-out-policy tools/relevance-eval/policies/cosine-v1.json and use a new held-out report name. Raw lexical, semantic and production hybrid results are retained. scores.hypothesis_semantic evaluates raw semantic candidates after the frozen cutoff; it is not a newly executed filtered hybrid fusion. This diagnostic does not deploy a threshold into the shared engine. Do not retune this policy using the held-out outcomes.

Supply an existing output directory and a new output filename. Existing evidence is never overwritten: the output is reserved with create_new before costly work and subsequent phase/row checkpoints replace JSON atomically. The runner verifies fixture SHA-256, exact source bytes, and both pinned Google assets before indexing. It records runner/worker/model hashes, actual worker capabilities, indexing/startup timings, all lexical candidates (including match kinds and source locations), raw semantic cosine scores, and final hybrid results with detailed engine traces. Code intents run both All files and Code scope; all lanes use the same strict scope filters.

The final 20-result hybrid request uses an 80-candidate budget. The diagnostic therefore records 80 lexical/semantic candidates separately. Semantic candidates are the production routed, per-file lane, not all passage vectors. Hybrid scores are reciprocal-rank-fusion scores, not confidence. No arbitrary cosine cutoff is applied, no vectors are simulated, and no quality acceptance claim follows from a successful run.

Queries execute lexical → semantic → hybrid in a retained query-role session. Hybrid consequently reuses the preceding semantic query vector. These timings are ordered diagnostic timings, not cold first input, UI latency, independent warm p95, or production performance evidence. Partial completed rows are saved after each request; an interrupted/failed report remains complete: false.

This corpus is small: fictional customer documents, two real source files and independent AI annotation. It does not cover media or production scale. Preserve labels and original raw output. A later policy experiment must fix all routing, abstention and parameter choices on calibration only before exposing held-out queries, and must retain exact filename priority and source provenance.