searchi/tools/rerank-eval/README.md

Local cross-encoder diagnostic

This standalone research tool never modifies production search or its Google EmbeddingGemma 2 model. Do not confuse the cross-encoder's name with the historical MiniLM embedding experiment. Publisher weights stay under ignored .searchi data.

Pinned publisher revision, asset bytes/SHA-256 and licence provenance are in model-manifest.json; the pre-inference contract is in compatibility-plan.json. The existing CPU Python environment can execute it without installing Torch or changing production dependencies:

python tools/rerank-eval/download.py
$env:SEARCHI_FORCE_CPU = '1'
& tools/semantic-proof/.venv/Scripts/python.exe tools/rerank-eval/compatibility.py --output .searchi/rerank-eval/compatibility-v1.json

The runner verifies all assets, rejects changed frozen inputs, enables only CPU Execution Provider with one thread, bounds five pairs to 256 tokens including special tokens and at most 64 query tokens, checks finite one-logit-per-pair outputs, and records serial/batch parity and twenty warm timing samples. It creates new reports atomically and refuses overwriting prior evidence. A failed parity contract writes the observed report then exits nonzero. Never weaken its frozen tolerance after observing output.

The first execution failed serial/batch logit parity; see recorded result. It is not a product integration, relevance calibration or full search performance benchmark. Scores are not probabilities that the requested artifact exists. Any further experimental input or scoring policy needs a separate version and output path.

The separate unquantized follow-up preserves the first failed evidence:

python tools/rerank-eval/download.py --reference
$env:SEARCHI_FORCE_CPU = '1'
& tools/semantic-proof/.venv/Scripts/python.exe tools/rerank-eval/reference.py --output .searchi/rerank-eval/reference-v1.json

reference-plan.json retains five short pairs and adds a 256-token budget case. F32 parity passed; quantized score differences persisted. Mixed-length padded batching was slower. Neither result establishes general absence calibration or sufficient default-search latency. Detailed costs and possible VM contention are recorded in the linked report documentation.