searchi/docs/LOCAL-RERANKING.md

# Local reranking experiment

Status: standalone CPU compatibility diagnostic executed; production integration
and abstention remain unimplemented. Google EmbeddingGemma 2
remains the mandatory document/code/image/audio/video retrieval model. Ordinary
search remains local; this proposal adds no hosted inference or automatic downloads.
Research checked 2026-10-13 using publisher metadata and the pinned native source.

## Smallest candidate worth testing

Try `cross-encoder/mmarco-mMiniLMv2-L12-H384-v1` as a **separate text reranker**.
Its publisher identifies an Apache-2.0 multilingual query/passage classifier,
trained on translated mMARCO. The dataset explicitly includes English and German.
It does not replace Google embeddings or create another searchable vector index.
[Publisher model card](https://huggingface.co/cross-encoder/mmarco-mMiniLMv2-L12-H384-v1),
[dataset publisher](https://huggingface.co/datasets/unicamp-dl/mmarco).

The observed repository revision is
`1427fd652930e4ba29e8149678df786c240d8825`. Its small configuration files report
`XLMRobertaForSequenceClassification`, 12 layers, hidden size 384, vocabulary
250002, one output label and tokenizer maximum length 512. Metadata was read;
no model weights were downloaded.
[Configuration](https://huggingface.co/cross-encoder/mmarco-mMiniLMv2-L12-H384-v1/blob/1427fd652930e4ba29e8149678df786c240d8825/config.json),
[tokenizer configuration](https://huggingface.co/cross-encoder/mmarco-mMiniLMv2-L12-H384-v1/blob/1427fd652930e4ba29e8149678df786c240d8825/tokenizer_config.json).

Publisher artifacts provide an honest size comparison:

| Candidate | Publisher evidence | Disposition |
| --- | --- | --- |
| mMARCO multilingual MiniLM cross-encoder | FP32 safetensors 471 MB; SHA-256 `5daeca2481a76b5976a2bdc32f0a78532b6716da4f8cd3ff59460ef8d2f359b4` | First compatibility/cost experiment |
| Same model's quantized ONNX AVX2 export | 119 MB; SHA-256 `6c2513767fb63d008a4377bef7a7a3555433d9436342bb53e35a3a72ffc52d4b` | Shows compact export is possible; adds another runtime if adopted |
| BAAI bge-reranker-v2-m3 | Apache-2.0, FP32 safetensors 2.27 GB | Larger fallback research candidate; not the smallest first experiment |
| Jina reranker v2 multilingual | Publisher restricts repository to CC-BY-NC-4.0 | Do not select for a broadly distributable default |

Artifact sources: [MiniLM safetensors](https://huggingface.co/cross-encoder/mmarco-mMiniLMv2-L12-H384-v1/blob/1427fd652930e4ba29e8149678df786c240d8825/model.safetensors),
[MiniLM ONNX export](https://huggingface.co/cross-encoder/mmarco-mMiniLMv2-L12-H384-v1/blob/1427fd652930e4ba29e8149678df786c240d8825/onnx/model_quint8_avx2.onnx),
[BAAI artifact](https://huggingface.co/BAAI/bge-reranker-v2-m3/blob/324cc40576b08b305b9c65a867c26c173a477ae2/model.safetensors),
[Jina publisher licence](https://huggingface.co/jinaai/jina-reranker-v2-base-multilingual).
The ONNX export names target different instruction families; no universal CPU
compatibility or measured latency follows from the file names. Converted GGUF
size, quantization fidelity and peak memory are still unknown.

## Reuse the native runtime first

The existing llama source revision
`4d756bc72bf00a4aacf410ae15a2d315f3db400d` registers the candidate's actual
architecture in `conversion/bert.py`. The embedding example includes query/document
pair tokenization and `LLAMA_POOLING_TYPE_RANK`, with classification scores returned
as rank outputs. Upstream documents a reranking endpoint and native CPU operation.
This is source-level feasibility, **not a successful candidate conversion or execution**.
[Native conversion source](https://github.com/ggml-org/llama.cpp/blob/4d756bc72bf00a4aacf410ae15a2d315f3db400d/conversion/bert.py),
[native ranking example](https://github.com/ggml-org/llama.cpp/blob/4d756bc72bf00a4aacf410ae15a2d315f3db400d/examples/embedding/embedding.cpp),
[upstream server contract](https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md).

Searchi's current worker intentionally validates 768-dimensional mean-pooled
Google vectors. It cannot load this one-output classifier through its existing
operations. Keep that invariant. First use the pinned upstream ranking example
as a diagnostic rather than expanding the production worker protocol prematurely.
The no-logits patch avoids unused vocabulary outputs for noncausal embedding
contexts; classification-head rank outputs still need an actual compatibility test.
Removing the patch does not turn EmbeddingGemma into an answer verifier.

Convert publisher-pinned weights with the pinned converter, without remote custom
code. Record all tokenizer files, licence, sizes, SHA-256 values and conversion
commands. Begin with F16 comparison, then Q8 only if reference parity is acceptable.
Do not accept an unverified third-party GGUF merely because it loads. CPU execution
and batch-versus-serial agreement must precede packaging or GPU claims.

## Bounded diagnostic, before product integration

1. Preserve Google candidate retrieval and all strict filters. Take at most five
   distinct source-verified text passages, one per selected file, with citations.
   Preserve exact identifiers and filename priority separately.
2. Bound each query/passage pair to 256 tokenizer tokens, including special tokens;
   reserve at most 64 for the query and use actual token-aware passage truncation.
   Record truncated candidates. Do not treat truncated evidence as proof of absence.
3. Compare serial one-thread CPU scoring with a bounded five-pair batch. Reuse one
   warmed session. Record pair construction, tokenization, inference and ordering
   separately; retain first-call, warm p95 and memory measurements.
4. Predeclare a cost target: added warm p95 at most 200 ms and total warm search p95
   at most 500 ms on the declared reference machine. These are acceptance targets,
   not measurements. If unmet, keep the experiment out of default search.
5. During diagnostics never suppress results. A cross-encoder's logit, or its
   sigmoid, is not a calibrated probability that the requested file exists.
   Failure, timeout, missing model or invalid output must preserve ordinary search.

A text classifier can evaluate documents, code, genuine tags and verified speech
transcripts. It cannot determine whether a photograph contains an object or an
instrumental recording contains a violin from placeholder media snippets. Visual
and instrumental abstention require separate real multimodal evidence; this proposal
does not close those gaps.

## Independent calibration and release decision

The old 0.70 cosine hypothesis failed held-out positive recall, and a correct
German result scored below an absent request. Do not retune it or use the inspected
mixed/relevance-v2 results as unseen validation. See [recorded failure](RELEVANCE-CALIBRATION.md).

Freeze a new small source corpus before scoring, with separate topic/repository
families for calibration and held-out. Include English/German requests for present
artifacts, absent concrete artifacts with closely related neighbours, exploratory
queries, code implementations and negated/contradictory passages. Keep translated
intent pairs together. Freeze labels and source bytes, obtain independent review,
and record all Google raw candidates before reranking.

On calibration only, assess whether classifier scores plus retained retrieval
signals improve false-match handling without losing weak-but-correct results.
Predeclare any proposed suppression rule and its parameters, then evaluate held-out
once. Report positive recall, absent-query return rate, hard negatives, truncation
and latency together. A failed held-out policy is rejected, not adjusted on that
same held-out set. Until evidence supports a default rule, keep related suggestions
available and avoid a definitive no-match claim based solely on score.

## Actual CPU compatibility result

The native conversion route required absent Torch/Transformers dependencies.
Instead of installing them, the existing CPU-only diagnostic environment provided
ONNX Runtime 1.31.0, tokenizers 0.23.3 and NumPy 2.5.4 under Python 3.14.8. The
publisher's pinned 118620016-byte ONNX export, 17082660-byte tokenizer and bounded
configuration/card/licence files were downloaded and SHA-256 verified. No FP32
weights, native conversion or production dependencies were added.

[Standalone tool](../tools/rerank-eval/README.md) froze its contract before inference:
plan SHA-256 `02569e771ab74ff593842ff26d7eb42d1b0509bef09a1867cf0f1be6b1dc5a5a`,
manifest SHA-256 `92ec98e8ccb5f8e8e08fbf46c7a4eb1407479b4dd0fafaabf6dc7cfc7aae2da7`.
The five synthetic bilingual/text-code pairs are compatibility inputs, not an
independent relevance corpus. Inference used only `CPUExecutionProvider`, one
intra-op and one inter-op thread, sequential execution. The root agent's installer
compression and Google package-startup checks finished before this inference slot;
other unrelated VM load was not instrumented.

| Observation | Actual result |
| --- | --- |
| Asset verification / tokenizer setup | 93.30 / 506.61 ms |
| ONNX session startup | 314.88 ms |
| Pair token counts | 29, 37, 24, 27, 42 |
| First five serial pairs / first batch | 51.99 / 76.09 ms |
| Warm five-pair serial median / p95 | 52.42 / 70.93 ms |
| Warm batch p95 | 67.25 ms |
| Maximum serial/batch raw-logit difference | **0.880886** |
| Frozen parity tolerance | absolute 0.001 plus relative 0.001 |
| Disposition | **Parity failed; no deployment** |

Both score arrays are preserved; their ordering happens to agree on these five
pairs. That does not excuse the failed score-invariance contract, especially for
future rejection policies. Quantized batch effects are a possible explanation,
not a verified cause. Do not relax the tolerance from these results. No confidence
threshold was fitted or applied.

Ignored complete report `.searchi/rerank-eval/compatibility-v1.json`, SHA-256
`a72cb628e2b635fe3549219d6be3065b9d3153225fd602e76502822a9db24e7c`.
The process exited 1 after preserving this failed contract. No inference remains
running. Timings cover short synthetic pairs and omit source retrieval, verification,
admission, publication and UI costs; they do not prove the 256-token budget target
or whole-pipeline performance. Peak private/resident memory was not measured.

Next, investigate score stability with a separately frozen serial-only diagnostic
and an unquantized reference comparison before any calibration. Do not silently
switch the existing plan's batch policy or overwrite this failure. Native runtime
compatibility and Windows/Linux/macOS portability remain unverified for this model.


## Unquantized reference follow-up

A separately frozen experiment verified the publisher F32 ONNX export at the same
revision: 470883696 bytes, SHA-256
`3e9a03ed1e966f7c5288dd4230e3d6a9bf5e3a170a06f1f4241c5bca12c6487c`.
Its plan SHA-256 is
`174867d4212b222e5302b1b16832b371c6bd38ec991f4145c952ed35446341f1`;
manifest SHA-256 is
`3c252dfc10e03ebba59a7ff0e33e6e2183a26c71862c52f0700f27ba659142be`.
It retained the original five pairs and added one deliberately long pair, exercising
64 query tokens and the complete 256-token pair budget. The original failed
quantized report, tolerance and input plan were preserved.

F32 serial/batch parity **passed with zero observed logit difference** on these six
inputs. The quantized single-pair outputs differed from F32 by up to 0.734820;
ordering of the original five scores agreed. Quantized serial/batch differences
remained observable at 0.378412 with the extra long input. This narrows the observed
problem to the quantized export/execution path on this diagnostic; it does not
identify the precise graph operator or prove general quantization fidelity.

| Six-pair F32 diagnostic | Actual observation |
| --- | --- |
| Asset verification / tokenizer setup | 388.83 / 497.59 ms |
| Session startup | 1147.90 ms |
| First serial / padded batch | 205.91 / 723.01 ms |
| Warm serial median / p95 | 202.45 / 235.67 ms |
| Warm padded batch p95 | 1017.59 ms |

The mixed-length batch padded all six pairs to 256 tokens, whereas serial execution
retained each pair's actual length. These are different work volumes, not equivalent
hardware scaling. Neither measured result establishes five full-budget passages
under the proposed 200-ms target. The existing user app's Google query-worker
startup may have overlapped the initial reference execution; unrelated VM contention
was not instrumented. Treat these as diagnostic timings, not clean-room performance.

Ignored reference report `.searchi/rerank-eval/reference-v1.json`, SHA-256
`de1d44245d6eb5fbf2a427b08b756b7c271b4daee57436155fa79ce9f684f100`.
The process exited zero and owned inference ended. All executions explicitly used
only CPU provider and one thread. No suppression, independent absence calibration,
production integration, GPU or other-platform execution occurred.

The next useful experiment is new independent relevance calibration with deterministic
single-pair scoring, while keeping the original mixed search results available.
Default deployment still requires measured full-budget cost, memory, reference
fidelity and positive-recall preservation. The batch failure is a reason to avoid
using uncalibrated quantized batch logits for automatic no-answer decisions.