searchi/docs/LOCAL-RERANKING.md

Local reranking experiment

Status: standalone CPU compatibility diagnostic executed; production integration and abstention remain unimplemented. Google EmbeddingGemma 2 remains the mandatory document/code/image/audio/video retrieval model. Ordinary search remains local; this proposal adds no hosted inference or automatic downloads. Research checked 2026-10-13 using publisher metadata and the pinned native source.

Smallest candidate worth testing

Try cross-encoder/mmarco-mMiniLMv2-L12-H384-v1 as a separate text reranker. Its publisher identifies an Apache-2.0 multilingual query/passage classifier, trained on translated mMARCO. The dataset explicitly includes English and German. It does not replace Google embeddings or create another searchable vector index. Publisher model card, dataset publisher.

The observed repository revision is 1427fd652930e4ba29e8149678df786c240d8825. Its small configuration files report XLMRobertaForSequenceClassification, 12 layers, hidden size 384, vocabulary 250002, one output label and tokenizer maximum length 512. Metadata was read; no model weights were downloaded. Configuration, tokenizer configuration.

Publisher artifacts provide an honest size comparison:

Candidate	Publisher evidence	Disposition
mMARCO multilingual MiniLM cross-encoder	FP32 safetensors 471 MB; SHA-256 5daeca2481a76b5976a2bdc32f0a78532b6716da4f8cd3ff59460ef8d2f359b4	First compatibility/cost experiment
Same model's quantized ONNX AVX2 export	119 MB; SHA-256 6c2513767fb63d008a4377bef7a7a3555433d9436342bb53e35a3a72ffc52d4b	Shows compact export is possible; adds another runtime if adopted
BAAI bge-reranker-v2-m3	Apache-2.0, FP32 safetensors 2.27 GB	Larger fallback research candidate; not the smallest first experiment
Jina reranker v2 multilingual	Publisher restricts repository to CC-BY-NC-4.0	Do not select for a broadly distributable default

Artifact sources: MiniLM safetensors, MiniLM ONNX export, BAAI artifact, Jina publisher licence. The ONNX export names target different instruction families; no universal CPU compatibility or measured latency follows from the file names. Converted GGUF size, quantization fidelity and peak memory are still unknown.

Reuse the native runtime first

The existing llama source revision 4d756bc72bf00a4aacf410ae15a2d315f3db400d registers the candidate's actual architecture in conversion/bert.py. The embedding example includes query/document pair tokenization and LLAMA_POOLING_TYPE_RANK, with classification scores returned as rank outputs. Upstream documents a reranking endpoint and native CPU operation. This is source-level feasibility, not a successful candidate conversion or execution. Native conversion source, native ranking example, upstream server contract.

Searchi's current worker intentionally validates 768-dimensional mean-pooled Google vectors. It cannot load this one-output classifier through its existing operations. Keep that invariant. First use the pinned upstream ranking example as a diagnostic rather than expanding the production worker protocol prematurely. The no-logits patch avoids unused vocabulary outputs for noncausal embedding contexts; classification-head rank outputs still need an actual compatibility test. Removing the patch does not turn EmbeddingGemma into an answer verifier.

Convert publisher-pinned weights with the pinned converter, without remote custom code. Record all tokenizer files, licence, sizes, SHA-256 values and conversion commands. Begin with F16 comparison, then Q8 only if reference parity is acceptable. Do not accept an unverified third-party GGUF merely because it loads. CPU execution and batch-versus-serial agreement must precede packaging or GPU claims.

Bounded diagnostic, before product integration

1. Preserve Google candidate retrieval and all strict filters. Take at most five distinct source-verified text passages, one per selected file, with citations. Preserve exact identifiers and filename priority separately.
2. Bound each query/passage pair to 256 tokenizer tokens, including special tokens; reserve at most 64 for the query and use actual token-aware passage truncation. Record truncated candidates. Do not treat truncated evidence as proof of absence.
3. Compare serial one-thread CPU scoring with a bounded five-pair batch. Reuse one warmed session. Record pair construction, tokenization, inference and ordering separately; retain first-call, warm p95 and memory measurements.
4. Predeclare a cost target: added warm p95 at most 200 ms and total warm search p95 at most 500 ms on the declared reference machine. These are acceptance targets, not measurements. If unmet, keep the experiment out of default search.
5. During diagnostics never suppress results. A cross-encoder's logit, or its sigmoid, is not a calibrated probability that the requested file exists. Failure, timeout, missing model or invalid output must preserve ordinary search.

A text classifier can evaluate documents, code, genuine tags and verified speech transcripts. It cannot determine whether a photograph contains an object or an instrumental recording contains a violin from placeholder media snippets. Visual and instrumental abstention require separate real multimodal evidence; this proposal does not close those gaps.

Independent calibration and release decision

The old 0.70 cosine hypothesis failed held-out positive recall, and a correct German result scored below an absent request. Do not retune it or use the inspected mixed/relevance-v2 results as unseen validation. See recorded failure.

Freeze a new small source corpus before scoring, with separate topic/repository families for calibration and held-out. Include English/German requests for present artifacts, absent concrete artifacts with closely related neighbours, exploratory queries, code implementations and negated/contradictory passages. Keep translated intent pairs together. Freeze labels and source bytes, obtain independent review, and record all Google raw candidates before reranking.

On calibration only, assess whether classifier scores plus retained retrieval signals improve false-match handling without losing weak-but-correct results. Predeclare any proposed suppression rule and its parameters, then evaluate held-out once. Report positive recall, absent-query return rate, hard negatives, truncation and latency together. A failed held-out policy is rejected, not adjusted on that same held-out set. Until evidence supports a default rule, keep related suggestions available and avoid a definitive no-match claim based solely on score.

Actual CPU compatibility result

The native conversion route required absent Torch/Transformers dependencies. Instead of installing them, the existing CPU-only diagnostic environment provided ONNX Runtime 1.31.0, tokenizers 0.23.3 and NumPy 2.5.4 under Python 3.14.8. The publisher's pinned 118620016-byte ONNX export, 17082660-byte tokenizer and bounded configuration/card/licence files were downloaded and SHA-256 verified. No FP32 weights, native conversion or production dependencies were added.

Standalone tool froze its contract before inference: plan SHA-256 02569e771ab74ff593842ff26d7eb42d1b0509bef09a1867cf0f1be6b1dc5a5a, manifest SHA-256 92ec98e8ccb5f8e8e08fbf46c7a4eb1407479b4dd0fafaabf6dc7cfc7aae2da7. The five synthetic bilingual/text-code pairs are compatibility inputs, not an independent relevance corpus. Inference used only CPUExecutionProvider, one intra-op and one inter-op thread, sequential execution. The root agent's installer compression and Google package-startup checks finished before this inference slot; other unrelated VM load was not instrumented.

Observation	Actual result
Asset verification / tokenizer setup	93.30 / 506.61 ms
ONNX session startup	314.88 ms
Pair token counts	29, 37, 24, 27, 42
First five serial pairs / first batch	51.99 / 76.09 ms
Warm five-pair serial median / p95	52.42 / 70.93 ms
Warm batch p95	67.25 ms
Maximum serial/batch raw-logit difference	0.880886
Frozen parity tolerance	absolute 0.001 plus relative 0.001
Disposition	Parity failed; no deployment

Both score arrays are preserved; their ordering happens to agree on these five pairs. That does not excuse the failed score-invariance contract, especially for future rejection policies. Quantized batch effects are a possible explanation, not a verified cause. Do not relax the tolerance from these results. No confidence threshold was fitted or applied.

Ignored complete report .searchi/rerank-eval/compatibility-v1.json, SHA-256 a72cb628e2b635fe3549219d6be3065b9d3153225fd602e76502822a9db24e7c. The process exited 1 after preserving this failed contract. No inference remains running. Timings cover short synthetic pairs and omit source retrieval, verification, admission, publication and UI costs; they do not prove the 256-token budget target or whole-pipeline performance. Peak private/resident memory was not measured.

Next, investigate score stability with a separately frozen serial-only diagnostic and an unquantized reference comparison before any calibration. Do not silently switch the existing plan's batch policy or overwrite this failure. Native runtime compatibility and Windows/Linux/macOS portability remain unverified for this model.

Unquantized reference follow-up

A separately frozen experiment verified the publisher F32 ONNX export at the same revision: 470883696 bytes, SHA-256 3e9a03ed1e966f7c5288dd4230e3d6a9bf5e3a170a06f1f4241c5bca12c6487c. Its plan SHA-256 is 174867d4212b222e5302b1b16832b371c6bd38ec991f4145c952ed35446341f1; manifest SHA-256 is 3c252dfc10e03ebba59a7ff0e33e6e2183a26c71862c52f0700f27ba659142be. It retained the original five pairs and added one deliberately long pair, exercising 64 query tokens and the complete 256-token pair budget. The original failed quantized report, tolerance and input plan were preserved.

F32 serial/batch parity passed with zero observed logit difference on these six inputs. The quantized single-pair outputs differed from F32 by up to 0.734820; ordering of the original five scores agreed. Quantized serial/batch differences remained observable at 0.378412 with the extra long input. This narrows the observed problem to the quantized export/execution path on this diagnostic; it does not identify the precise graph operator or prove general quantization fidelity.

Six-pair F32 diagnostic	Actual observation
Asset verification / tokenizer setup	388.83 / 497.59 ms
Session startup	1147.90 ms
First serial / padded batch	205.91 / 723.01 ms
Warm serial median / p95	202.45 / 235.67 ms
Warm padded batch p95	1017.59 ms

The mixed-length batch padded all six pairs to 256 tokens, whereas serial execution retained each pair's actual length. These are different work volumes, not equivalent hardware scaling. Neither measured result establishes five full-budget passages under the proposed 200-ms target. The existing user app's Google query-worker startup may have overlapped the initial reference execution; unrelated VM contention was not instrumented. Treat these as diagnostic timings, not clean-room performance.

Ignored reference report .searchi/rerank-eval/reference-v1.json, SHA-256 de1d44245d6eb5fbf2a427b08b756b7c271b4daee57436155fa79ce9f684f100. The process exited zero and owned inference ended. All executions explicitly used only CPU provider and one thread. No suppression, independent absence calibration, production integration, GPU or other-platform execution occurred.

The next useful experiment is new independent relevance calibration with deterministic single-pair scoring, while keeping the original mixed search results available. Default deployment still requires measured full-budget cost, memory, reference fidelity and positive-recall preservation. The batch failure is a reason to avoid using uncalibrated quantized batch logits for automatic no-answer decisions.