50 lines
2.8 KiB
Markdown
50 lines
2.8 KiB
Markdown
|
# Calibration-only relevance diagnostic
|
||
|
|
|
||
|
|
This internal runner uses searchi's shared Rust engine and genuine Google
|
||
|
|
EmbeddingGemma 2. It has **no held-out execution option**. It indexes all 14 frozen
|
||
|
|
source files as the fixed distractor library, but evaluates only the 24
|
||
|
|
calibration queries. Held-out queries stay unevaluated until a policy and its
|
||
|
|
parameters are frozen using calibration evidence.
|
||
|
|
|
||
|
|
First verify all frozen labels, source bytes, licenses and split integrity:
|
||
|
|
|
||
|
|
```powershell
|
||
|
|
python fixtures/relevance-v2/verify.py
|
||
|
|
cargo test --manifest-path tools/relevance-eval/Cargo.toml --locked
|
||
|
|
cargo build --release --manifest-path tools/relevance-eval/Cargo.toml --locked
|
||
|
|
```
|
||
|
|
|
||
|
|
Only run when the shared CPU inference slot is free:
|
||
|
|
|
||
|
|
```powershell
|
||
|
|
$env:SEARCHI_FORCE_CPU = '1'
|
||
|
|
tools/relevance-eval/target/release/searchi-relevance-eval.exe fixtures/relevance-v2 C:/path/to/verified-models C:/path/to/searchi-gemma-worker.exe .searchi/qa/calibration-v2.json
|
||
|
|
```
|
||
|
|
|
||
|
|
Supply an existing output directory and a new output filename. Existing evidence
|
||
|
|
is never overwritten: the output is reserved with `create_new` before costly
|
||
|
|
work and subsequent phase/row checkpoints replace JSON atomically. The runner verifies fixture SHA-256, exact source bytes,
|
||
|
|
and both pinned Google assets before indexing. It records runner/worker/model
|
||
|
|
hashes, actual worker capabilities, indexing/startup timings, all lexical
|
||
|
|
candidates (including match kinds and source locations), raw semantic cosine
|
||
|
|
scores, and final hybrid results with detailed engine traces. Code intents run
|
||
|
|
both All files and Code scope; all lanes use the same strict scope filters.
|
||
|
|
|
||
|
|
The final 20-result hybrid request uses an 80-candidate budget. The diagnostic
|
||
|
|
therefore records 80 lexical/semantic candidates separately. Semantic candidates
|
||
|
|
are the production routed, per-file lane, not all passage vectors. Hybrid scores
|
||
|
|
are reciprocal-rank-fusion scores, **not confidence**. No arbitrary cosine cutoff
|
||
|
|
is applied, no vectors are simulated, and no quality acceptance claim follows
|
||
|
|
from a successful run.
|
||
|
|
|
||
|
|
Queries execute lexical → semantic → hybrid in a retained query-role session.
|
||
|
|
Hybrid consequently reuses the preceding semantic query vector. These timings
|
||
|
|
are ordered diagnostic timings, not cold first input, UI latency, independent
|
||
|
|
warm p95, or production performance evidence. Partial completed rows are saved
|
||
|
|
after each request; an interrupted/failed report remains `complete: false`.
|
||
|
|
|
||
|
|
This corpus is small: fictional customer documents, two real source files and
|
||
|
|
independent AI annotation. It does not cover media or production scale. Preserve
|
||
|
|
labels and original raw output. A later policy experiment must fix all routing,
|
||
|
|
abstention and parameter choices on calibration only before exposing held-out
|
||
|
|
queries, and must retain exact filename priority and source provenance.
|