Add shared-engine calibration-only relevance diagnostics

tools/relevance-eval/README.md

@@ -0,0 +1,50 @@
+# Calibration-only relevance diagnostic
+
+This internal runner uses searchi's shared Rust engine and genuine Google
+EmbeddingGemma 2. It has **no held-out execution option**. It indexes all 14 frozen
+source files as the fixed distractor library, but evaluates only the 24
+calibration queries. Held-out queries stay unevaluated until a policy and its
+parameters are frozen using calibration evidence.
+
+First verify all frozen labels, source bytes, licenses and split integrity:
+
+```powershell
+python fixtures/relevance-v2/verify.py
+cargo test --manifest-path tools/relevance-eval/Cargo.toml --locked
+cargo build --release --manifest-path tools/relevance-eval/Cargo.toml --locked
+```
+
+Only run when the shared CPU inference slot is free:
+
+```powershell
+$env:SEARCHI_FORCE_CPU = '1'
+tools/relevance-eval/target/release/searchi-relevance-eval.exe fixtures/relevance-v2 C:/path/to/verified-models C:/path/to/searchi-gemma-worker.exe .searchi/qa/calibration-v2.json
+```
+
+Supply an existing output directory and a new output filename. Existing evidence
+is never overwritten: the output is reserved with `create_new` before costly
+work and subsequent phase/row checkpoints replace JSON atomically. The runner verifies fixture SHA-256, exact source bytes,
+and both pinned Google assets before indexing. It records runner/worker/model
+hashes, actual worker capabilities, indexing/startup timings, all lexical
+candidates (including match kinds and source locations), raw semantic cosine
+scores, and final hybrid results with detailed engine traces. Code intents run
+both All files and Code scope; all lanes use the same strict scope filters.
+
+The final 20-result hybrid request uses an 80-candidate budget. The diagnostic
+therefore records 80 lexical/semantic candidates separately. Semantic candidates
+are the production routed, per-file lane, not all passage vectors. Hybrid scores
+are reciprocal-rank-fusion scores, **not confidence**. No arbitrary cosine cutoff
+is applied, no vectors are simulated, and no quality acceptance claim follows
+from a successful run.
+
+Queries execute lexical → semantic → hybrid in a retained query-role session.
+Hybrid consequently reuses the preceding semantic query vector. These timings
+are ordered diagnostic timings, not cold first input, UI latency, independent
+warm p95, or production performance evidence. Partial completed rows are saved
+after each request; an interrupted/failed report remains `complete: false`.
+
+This corpus is small: fictional customer documents, two real source files and
+independent AI annotation. It does not cover media or production scale. Preserve
+labels and original raw output. A later policy experiment must fix all routing,
+abstention and parameter choices on calibration only before exposing held-out
+queries, and must retain exact filename priority and source provenance.