44 lines
2.3 KiB
Markdown
44 lines
2.3 KiB
Markdown
|
# Local cross-encoder diagnostic
|
||
|
|
|
||
|
|
This standalone research tool never modifies production search or its Google
|
||
|
|
EmbeddingGemma 2 model. Do not confuse the cross-encoder's name with the historical
|
||
|
|
MiniLM embedding experiment. Publisher weights stay under ignored `.searchi` data.
|
||
|
|
|
||
|
|
Pinned publisher revision, asset bytes/SHA-256 and licence provenance are in
|
||
|
|
`model-manifest.json`; the pre-inference contract is in `compatibility-plan.json`.
|
||
|
|
The existing CPU Python environment can execute it without installing Torch or
|
||
|
|
changing production dependencies:
|
||
|
|
|
||
|
|
```powershell
|
||
|
|
python tools/rerank-eval/download.py
|
||
|
|
$env:SEARCHI_FORCE_CPU = '1'
|
||
|
|
& tools/semantic-proof/.venv/Scripts/python.exe tools/rerank-eval/compatibility.py --output .searchi/rerank-eval/compatibility-v1.json
|
||
|
|
```
|
||
|
|
|
||
|
|
The runner verifies all assets, rejects changed frozen inputs, enables only CPU
|
||
|
|
Execution Provider with one thread, bounds five pairs to 256 tokens including
|
||
|
|
special tokens and at most 64 query tokens, checks finite one-logit-per-pair
|
||
|
|
outputs, and records serial/batch parity and twenty warm timing samples. It creates
|
||
|
|
new reports atomically and refuses overwriting prior evidence. A failed parity
|
||
|
|
contract writes the observed report then exits nonzero. Never weaken its frozen
|
||
|
|
tolerance after observing output.
|
||
|
|
|
||
|
|
The first execution failed serial/batch logit parity; see
|
||
|
|
[recorded result](../../docs/LOCAL-RERANKING.md). It is not a product integration,
|
||
|
|
relevance calibration or full search performance benchmark. Scores are not
|
||
|
|
probabilities that the requested artifact exists. Any further experimental input
|
||
|
|
or scoring policy needs a separate version and output path.
|
||
|
|
|
||
|
|
The separate unquantized follow-up preserves the first failed evidence:
|
||
|
|
|
||
|
|
```powershell
|
||
|
|
python tools/rerank-eval/download.py --reference
|
||
|
|
$env:SEARCHI_FORCE_CPU = '1'
|
||
|
|
& tools/semantic-proof/.venv/Scripts/python.exe tools/rerank-eval/reference.py --output .searchi/rerank-eval/reference-v1.json
|
||
|
|
```
|
||
|
|
|
||
|
|
`reference-plan.json` retains five short pairs and adds a 256-token budget case.
|
||
|
|
F32 parity passed; quantized score differences persisted. Mixed-length padded
|
||
|
|
batching was slower. Neither result establishes general absence calibration or
|
||
|
|
sufficient default-search latency. Detailed costs and possible VM contention are
|
||
|
|
recorded in the linked report documentation.
|