This internal runner uses searchi's shared Rust engine and genuine Google EmbeddingGemma 2. It has no held-out execution option. It indexes all 14 frozen source files as the fixed distractor library, but evaluates only the 24 calibration queries. Held-out queries stay unevaluated until a policy and its parameters are frozen using calibration evidence.

First verify all frozen labels, source bytes, licenses and split integrity:

python fixtures/relevance-v2/verify.py
cargo test --manifest-path tools/relevance-eval/Cargo.toml --locked
cargo build --release --manifest-path tools/relevance-eval/Cargo.toml --locked

Only run when the shared CPU inference slot is free:

$env:SEARCHI_FORCE_CPU = '1'
tools/relevance-eval/target/release/searchi-relevance-eval.exe fixtures/relevance-v2 C:/path/to/verified-models C:/path/to/searchi-gemma-worker.exe .searchi/qa/calibration-v2.json

Supply an existing output directory and a new output filename. Existing evidence is never overwritten: the output is reserved with create_new before costly work and subsequent phase/row checkpoints replace JSON atomically. The runner verifies fixture SHA-256, exact source bytes, and both pinned Google assets before indexing. It records runner/worker/model hashes, actual worker capabilities, indexing/startup timings, all lexical candidates (including match kinds and source locations), raw semantic cosine scores, and final hybrid results with detailed engine traces. Code intents run both All files and Code scope; all lanes use the same strict scope filters.

The final 20-result hybrid request uses an 80-candidate budget. The diagnostic therefore records 80 lexical/semantic candidates separately. Semantic candidates are the production routed, per-file lane, not all passage vectors. Hybrid scores are reciprocal-rank-fusion scores, not confidence. No arbitrary cosine cutoff is applied, no vectors are simulated, and no quality acceptance claim follows from a successful run.

Queries execute lexical → semantic → hybrid in a retained query-role session. Hybrid consequently reuses the preceding semantic query vector. These timings are ordered diagnostic timings, not cold first input, UI latency, independent warm p95, or production performance evidence. Partial completed rows are saved after each request; an interrupted/failed report remains complete: false.

This corpus is small: fictional customer documents, two real source files and independent AI annotation. It does not cover media or production scale. Preserve labels and original raw output. A later policy experiment must fix all routing, abstention and parameter choices on calibration only before exposing held-out queries, and must retain exact filename priority and source provenance.