60 lines
3.3 KiB
Markdown
60 lines
3.3 KiB
Markdown
# Frozen relevance diagnostic
|
|
|
|
This internal runner uses searchi's shared Rust engine and genuine Google
|
|
EmbeddingGemma 2. By default it indexes all 14 frozen source files as the fixed
|
|
distractor library and evaluates only the 24 calibration queries. Held-out
|
|
execution requires `--held-out-policy` with the exact SHA-256 of the hypothesis
|
|
frozen in commit `e7245fe`, before held-out inference. Changed policy bytes are
|
|
rejected. The two splits are selected independently and never mixed in one run.
|
|
|
|
First verify all frozen labels, source bytes, licenses and split integrity:
|
|
|
|
```powershell
|
|
python fixtures/relevance-v2/verify.py
|
|
cargo test --manifest-path tools/relevance-eval/Cargo.toml --locked
|
|
cargo build --release --manifest-path tools/relevance-eval/Cargo.toml --locked
|
|
```
|
|
|
|
Only run when the shared CPU inference slot is free:
|
|
|
|
```powershell
|
|
$env:SEARCHI_FORCE_CPU = '1'
|
|
tools/relevance-eval/target/release/searchi-relevance-eval.exe fixtures/relevance-v2 C:/path/to/verified-models C:/path/to/searchi-gemma-worker.exe .searchi/qa/calibration-v2.json
|
|
```
|
|
|
|
For the one frozen held-out evaluation, append
|
|
`--held-out-policy tools/relevance-eval/policies/cosine-v1.json` and use a new
|
|
held-out report name. Raw lexical, semantic and production hybrid results are
|
|
retained. `scores.hypothesis_semantic` evaluates raw semantic candidates after
|
|
the frozen cutoff; it is not a newly executed filtered hybrid fusion. This
|
|
diagnostic does not deploy a threshold into the shared engine. Do not retune
|
|
this policy using the held-out outcomes.
|
|
|
|
Supply an existing output directory and a new output filename. Existing evidence
|
|
is never overwritten: the output is reserved with `create_new` before costly
|
|
work and subsequent phase/row checkpoints replace JSON atomically. The runner verifies fixture SHA-256, exact source bytes,
|
|
and both pinned Google assets before indexing. It records runner/worker/model
|
|
hashes, actual worker capabilities, indexing/startup timings, all lexical
|
|
candidates (including match kinds and source locations), raw semantic cosine
|
|
scores, and final hybrid results with detailed engine traces. Code intents run
|
|
both All files and Code scope; all lanes use the same strict scope filters.
|
|
|
|
The final 20-result hybrid request uses an 80-candidate budget. The diagnostic
|
|
therefore records 80 lexical/semantic candidates separately. Semantic candidates
|
|
are the production routed, per-file lane, not all passage vectors. Hybrid scores
|
|
are reciprocal-rank-fusion scores, **not confidence**. No arbitrary cosine cutoff
|
|
is applied, no vectors are simulated, and no quality acceptance claim follows
|
|
from a successful run.
|
|
|
|
Queries execute lexical → semantic → hybrid in a retained query-role session.
|
|
Hybrid consequently reuses the preceding semantic query vector. These timings
|
|
are ordered diagnostic timings, not cold first input, UI latency, independent
|
|
warm p95, or production performance evidence. Partial completed rows are saved
|
|
after each request; an interrupted/failed report remains `complete: false`.
|
|
|
|
This corpus is small: fictional customer documents, two real source files and
|
|
independent AI annotation. It does not cover media or production scale. Preserve
|
|
labels and original raw output. A later policy experiment must fix all routing,
|
|
abstention and parameter choices on calibration only before exposing held-out
|
|
queries, and must retain exact filename priority and source provenance.
|