karl/searchi
1
0 Fork 0
Code Issues Pull Requests Packages Projects Releases 10 Wiki Activity

Record unseen music regressions and actual local reranker fidelity and cost

Browse Source
This commit is contained in:
karl committed 2026-10-13 18:59:45 +02:00
1 parent cbf07b0575
commit ec73d453e3
11 files changed
+1665 -1
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

docs/LOCAL-RERANKING.md
+208
View File
@@ -0,0 +1,208 @@
# Local reranking experiment
Status: standalone CPU compatibility diagnostic executed; production integration
and abstention remain unimplemented. Google EmbeddingGemma 2
remains the mandatory document/code/image/audio/video retrieval model. Ordinary
search remains local; this proposal adds no hosted inference or automatic downloads.
Research checked 2026-10-13 using publisher metadata and the pinned native source.
## Smallest candidate worth testing
Try `cross-encoder/mmarco-mMiniLMv2-L12-H384-v1` as a **separate text reranker**.
Its publisher identifies an Apache-2.0 multilingual query/passage classifier,
trained on translated mMARCO. The dataset explicitly includes English and German.
It does not replace Google embeddings or create another searchable vector index.
[Publisher model card](https://huggingface.co/cross-encoder/mmarco-mMiniLMv2-L12-H384-v1),
[dataset publisher](https://huggingface.co/datasets/unicamp-dl/mmarco).
The observed repository revision is
`1427fd652930e4ba29e8149678df786c240d8825`. Its small configuration files report
`XLMRobertaForSequenceClassification`, 12 layers, hidden size 384, vocabulary
250002, one output label and tokenizer maximum length 512. Metadata was read;
no model weights were downloaded.
[Configuration](https://huggingface.co/cross-encoder/mmarco-mMiniLMv2-L12-H384-v1/blob/1427fd652930e4ba29e8149678df786c240d8825/config.json),
[tokenizer configuration](https://huggingface.co/cross-encoder/mmarco-mMiniLMv2-L12-H384-v1/blob/1427fd652930e4ba29e8149678df786c240d8825/tokenizer_config.json).
Publisher artifacts provide an honest size comparison:
| Candidate | Publisher evidence | Disposition |
| --- | --- | --- |
| mMARCO multilingual MiniLM cross-encoder | FP32 safetensors 471 MB; SHA-256 `5daeca2481a76b5976a2bdc32f0a78532b6716da4f8cd3ff59460ef8d2f359b4` | First compatibility/cost experiment |
| Same model's quantized ONNX AVX2 export | 119 MB; SHA-256 `6c2513767fb63d008a4377bef7a7a3555433d9436342bb53e35a3a72ffc52d4b` | Shows compact export is possible; adds another runtime if adopted |
| BAAI bge-reranker-v2-m3 | Apache-2.0, FP32 safetensors 2.27 GB | Larger fallback research candidate; not the smallest first experiment |
| Jina reranker v2 multilingual | Publisher restricts repository to CC-BY-NC-4.0 | Do not select for a broadly distributable default |
Artifact sources: [MiniLM safetensors](https://huggingface.co/cross-encoder/mmarco-mMiniLMv2-L12-H384-v1/blob/1427fd652930e4ba29e8149678df786c240d8825/model.safetensors),
[MiniLM ONNX export](https://huggingface.co/cross-encoder/mmarco-mMiniLMv2-L12-H384-v1/blob/1427fd652930e4ba29e8149678df786c240d8825/onnx/model_quint8_avx2.onnx),
[BAAI artifact](https://huggingface.co/BAAI/bge-reranker-v2-m3/blob/324cc40576b08b305b9c65a867c26c173a477ae2/model.safetensors),
[Jina publisher licence](https://huggingface.co/jinaai/jina-reranker-v2-base-multilingual).
The ONNX export names target different instruction families; no universal CPU
compatibility or measured latency follows from the file names. Converted GGUF
size, quantization fidelity and peak memory are still unknown.
## Reuse the native runtime first
The existing llama source revision
`4d756bc72bf00a4aacf410ae15a2d315f3db400d` registers the candidate's actual
architecture in `conversion/bert.py`. The embedding example includes query/document
pair tokenization and `LLAMA_POOLING_TYPE_RANK`, with classification scores returned
as rank outputs. Upstream documents a reranking endpoint and native CPU operation.
This is source-level feasibility, **not a successful candidate conversion or execution**.
[Native conversion source](https://github.com/ggml-org/llama.cpp/blob/4d756bc72bf00a4aacf410ae15a2d315f3db400d/conversion/bert.py),
[native ranking example](https://github.com/ggml-org/llama.cpp/blob/4d756bc72bf00a4aacf410ae15a2d315f3db400d/examples/embedding/embedding.cpp),
[upstream server contract](https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md).
Searchi's current worker intentionally validates 768-dimensional mean-pooled
Google vectors. It cannot load this one-output classifier through its existing
operations. Keep that invariant. First use the pinned upstream ranking example
as a diagnostic rather than expanding the production worker protocol prematurely.
The no-logits patch avoids unused vocabulary outputs for noncausal embedding
contexts; classification-head rank outputs still need an actual compatibility test.
Removing the patch does not turn EmbeddingGemma into an answer verifier.
Convert publisher-pinned weights with the pinned converter, without remote custom
code. Record all tokenizer files, licence, sizes, SHA-256 values and conversion
commands. Begin with F16 comparison, then Q8 only if reference parity is acceptable.
Do not accept an unverified third-party GGUF merely because it loads. CPU execution
and batch-versus-serial agreement must precede packaging or GPU claims.
## Bounded diagnostic, before product integration
1. Preserve Google candidate retrieval and all strict filters. Take at most five
distinct source-verified text passages, one per selected file, with citations.
Preserve exact identifiers and filename priority separately.
2. Bound each query/passage pair to 256 tokenizer tokens, including special tokens;
reserve at most 64 for the query and use actual token-aware passage truncation.
Record truncated candidates. Do not treat truncated evidence as proof of absence.
3. Compare serial one-thread CPU scoring with a bounded five-pair batch. Reuse one
warmed session. Record pair construction, tokenization, inference and ordering
separately; retain first-call, warm p95 and memory measurements.
4. Predeclare a cost target: added warm p95 at most 200 ms and total warm search p95
at most 500 ms on the declared reference machine. These are acceptance targets,
not measurements. If unmet, keep the experiment out of default search.
5. During diagnostics never suppress results. A cross-encoder's logit, or its
sigmoid, is not a calibrated probability that the requested file exists.
Failure, timeout, missing model or invalid output must preserve ordinary search.
A text classifier can evaluate documents, code, genuine tags and verified speech
transcripts. It cannot determine whether a photograph contains an object or an
instrumental recording contains a violin from placeholder media snippets. Visual
and instrumental abstention require separate real multimodal evidence; this proposal
does not close those gaps.
## Independent calibration and release decision
The old 0.70 cosine hypothesis failed held-out positive recall, and a correct
German result scored below an absent request. Do not retune it or use the inspected
mixed/relevance-v2 results as unseen validation. See [recorded failure](RELEVANCE-CALIBRATION.md).
Freeze a new small source corpus before scoring, with separate topic/repository
families for calibration and held-out. Include English/German requests for present
artifacts, absent concrete artifacts with closely related neighbours, exploratory
queries, code implementations and negated/contradictory passages. Keep translated
intent pairs together. Freeze labels and source bytes, obtain independent review,
and record all Google raw candidates before reranking.
On calibration only, assess whether classifier scores plus retained retrieval
signals improve false-match handling without losing weak-but-correct results.
Predeclare any proposed suppression rule and its parameters, then evaluate held-out
once. Report positive recall, absent-query return rate, hard negatives, truncation
and latency together. A failed held-out policy is rejected, not adjusted on that
same held-out set. Until evidence supports a default rule, keep related suggestions
available and avoid a definitive no-match claim based solely on score.
## Actual CPU compatibility result
The native conversion route required absent Torch/Transformers dependencies.
Instead of installing them, the existing CPU-only diagnostic environment provided
ONNX Runtime 1.31.0, tokenizers 0.23.3 and NumPy 2.5.4 under Python 3.14.8. The
publisher's pinned 118620016-byte ONNX export, 17082660-byte tokenizer and bounded
configuration/card/licence files were downloaded and SHA-256 verified. No FP32
weights, native conversion or production dependencies were added.
[Standalone tool](../tools/rerank-eval/README.md) froze its contract before inference:
plan SHA-256 `02569e771ab74ff593842ff26d7eb42d1b0509bef09a1867cf0f1be6b1dc5a5a`,
manifest SHA-256 `92ec98e8ccb5f8e8e08fbf46c7a4eb1407479b4dd0fafaabf6dc7cfc7aae2da7`.
The five synthetic bilingual/text-code pairs are compatibility inputs, not an
independent relevance corpus. Inference used only `CPUExecutionProvider`, one
intra-op and one inter-op thread, sequential execution. The root agent's installer
compression and Google package-startup checks finished before this inference slot;
other unrelated VM load was not instrumented.
| Observation | Actual result |
| --- | --- |
| Asset verification / tokenizer setup | 93.30 / 506.61 ms |
| ONNX session startup | 314.88 ms |
| Pair token counts | 29, 37, 24, 27, 42 |
| First five serial pairs / first batch | 51.99 / 76.09 ms |
| Warm five-pair serial median / p95 | 52.42 / 70.93 ms |
| Warm batch p95 | 67.25 ms |
| Maximum serial/batch raw-logit difference | **0.880886** |
| Frozen parity tolerance | absolute 0.001 plus relative 0.001 |
| Disposition | **Parity failed; no deployment** |
Both score arrays are preserved; their ordering happens to agree on these five
pairs. That does not excuse the failed score-invariance contract, especially for
future rejection policies. Quantized batch effects are a possible explanation,
not a verified cause. Do not relax the tolerance from these results. No confidence
threshold was fitted or applied.
Ignored complete report `.searchi/rerank-eval/compatibility-v1.json`, SHA-256
`a72cb628e2b635fe3549219d6be3065b9d3153225fd602e76502822a9db24e7c`.
The process exited 1 after preserving this failed contract. No inference remains
running. Timings cover short synthetic pairs and omit source retrieval, verification,
admission, publication and UI costs; they do not prove the 256-token budget target
or whole-pipeline performance. Peak private/resident memory was not measured.
Next, investigate score stability with a separately frozen serial-only diagnostic
and an unquantized reference comparison before any calibration. Do not silently
switch the existing plan's batch policy or overwrite this failure. Native runtime
compatibility and Windows/Linux/macOS portability remain unverified for this model.
## Unquantized reference follow-up
A separately frozen experiment verified the publisher F32 ONNX export at the same
revision: 470883696 bytes, SHA-256
`3e9a03ed1e966f7c5288dd4230e3d6a9bf5e3a170a06f1f4241c5bca12c6487c`.
Its plan SHA-256 is
`174867d4212b222e5302b1b16832b371c6bd38ec991f4145c952ed35446341f1`;
manifest SHA-256 is
`3c252dfc10e03ebba59a7ff0e33e6e2183a26c71862c52f0700f27ba659142be`.
It retained the original five pairs and added one deliberately long pair, exercising
64 query tokens and the complete 256-token pair budget. The original failed
quantized report, tolerance and input plan were preserved.
F32 serial/batch parity **passed with zero observed logit difference** on these six
inputs. The quantized single-pair outputs differed from F32 by up to 0.734820;
ordering of the original five scores agreed. Quantized serial/batch differences
remained observable at 0.378412 with the extra long input. This narrows the observed
problem to the quantized export/execution path on this diagnostic; it does not
identify the precise graph operator or prove general quantization fidelity.
| Six-pair F32 diagnostic | Actual observation |
| --- | --- |
| Asset verification / tokenizer setup | 388.83 / 497.59 ms |
| Session startup | 1147.90 ms |
| First serial / padded batch | 205.91 / 723.01 ms |
| Warm serial median / p95 | 202.45 / 235.67 ms |
| Warm padded batch p95 | 1017.59 ms |
The mixed-length batch padded all six pairs to 256 tokens, whereas serial execution
retained each pair's actual length. These are different work volumes, not equivalent
hardware scaling. Neither measured result establishes five full-budget passages
under the proposed 200-ms target. The existing user app's Google query-worker
startup may have overlapped the initial reference execution; unrelated VM contention
was not instrumented. Treat these as diagnostic timings, not clean-room performance.
Ignored reference report `.searchi/rerank-eval/reference-v1.json`, SHA-256
`de1d44245d6eb5fbf2a427b08b756b7c271b4daee57436155fa79ce9f684f100`.
The process exited zero and owned inference ended. All executions explicitly used
only CPU provider and one thread. No suppression, independent absence calibration,
production integration, GPU or other-platform execution occurred.
The next useful experiment is new independent relevance calibration with deterministic
single-pair scoring, while keeping the original mixed search results available.
Default deployment still requires measured full-budget cost, memory, reference
fidelity and positive-recall preservation. The batch failure is a reason to avoid
using uncalibrated quantized batch logits for automatic no-answer decisions.
fixtures/music-validation-v1/README.md
+54 -1
View File
@@ -84,4 +84,57 @@ lanes compare media-only, original tags, missing tags and misleading tags withou
tuned weights or cutoff. Report positive recall and no-answer failure separately. tuned weights or cutoff. Report positive recall and no-answer failure separately.
This direct cosine diagnostic is not production ANN/hybrid validation. This direct cosine diagnostic is not production ANN/hybrid validation.
The earlier prepared plan 8598312836f3c027b578f9f93ca1cd2d38d708bc0fd713fe174f10776934cbb2 was never inferred. Pre-inference review found that its verification decoded full float PCM while the production decoder uses millisecond trimming and s16 normalization. The corrected plan above matches the actual production decoder and compares only the whitelisted tag evidence; labels, source bytes, query texts and ranking hypothesis are unchanged. The earlier plan is preserved under its original ignored directory. The earlier prepared plan
`8598312836f3c027b578f9f93ca1cd2d38d708bc0fd713fe174f10776934cbb2`
was never inferred. Pre-inference review found that its verification decoded full
float PCM while the production decoder uses millisecond trimming and s16
normalization. The corrected plan above matches the actual production decoder
and compares only the whitelisted tag evidence; labels, source bytes, query texts
and ranking hypothesis are unchanged. The earlier plan is preserved under its
original ignored directory.
## Actual frozen CPU results
The corrected policy completed from source `cbf07b0` without changing labels,
competitors, weights or a cutoff. Independent read-only recomputation checked all
18 queries, all four lanes, ten distinct competitors per lane and sorted scores.
Missing-metadata rankings exactly equal the media-only rankings for every query.
| Direct file retrieval | Media only | Original tags | Missing tags | Misleading tags |
| --- | ---: | ---: | ---: | ---: |
| Correct first file, fourteen positives | 8/14 | 11/14 | 8/14 | 5/14 |
| Recall in first five | 14/14 | 14/14 | 14/14 | 14/14 |
| MRR | 0.726 | 0.869 | 0.726 | 0.570 |
| No-answer queries returning candidates, failure | 4/4 | 4/4 | 4/4 | 4/4 |
Original fields improve both cello and both soprano-sax queries. However, the
previously correct German acoustic-guitar query regresses from first to second:
an unrelated tenor-sax title scores 0.65889 against the correct guitar audio's
0.65581. Original tags leave both bucket-muted-trumpet queries incorrect. Misleading
fields additionally lose both previously correct tenor-sax first hits and preserve
the guitar regression. These losses are not hidden by aggregate gains or top-five
recall. Four absent queries still return all ten candidates; no abstention
improvement or rejection threshold follows.
Ten media encodes (eight audio sources and two photos) plus sixteen real tag-field
encodes took **88.09 seconds** after model loading. That timer includes actual
probing/decoding/source checks as well as inference; it excludes model startup and
query inference and cannot isolate metadata's additional cost. The Windows 11
10.0.26300 shared VM exposed eight logical CPUs and 16708012 KiB visible RAM on
an AMD Ryzen 7 7800X3D host. Actual Google inference used one CPU thread, with the
normal runtime forced to CPU. The installed app's warmed CPU query worker remained
idle; no root model requests ran concurrently in the reserved inference slot.
The native runtime continues to identify audio support as experimental.
The raw ignored report `.searchi/qa/music-validation-v1-canonical/results.json`
contains 267667 bytes, SHA-256
`69bcef5ffa58c0f23e5f34641256255bf15f54c836f75fb1aa878b4bd3df1309`.
[Recomputed summary](evaluation-summary.json) preserves every query's first file,
source kind/field, score and first relevant rank, plus worker/fixture hashes.
**Do not deploy the unconditional maximum policy as a universally better music
ranking.** It is promising with informative correct tags, but this independently
frozen sample exposes a real regression and sensitivity to misleading metadata.
No policy was retuned after seeing these scores. Production ANN/hybrid evidence,
more representative music, human listening, missing-content calibration and
source-specific confidence remain open. The experimental media rating is unchanged.
fixtures/music-validation-v1/evaluation-summary.json
+844
View File
@@ -0,0 +1,844 @@
{
"source_commit": "cbf07b0",
"fixture_sha256": "f9d7785ff937020b802e6c56252e64bbc3cd22e9a09709ddce9b82f9adc1b577",
"prepared_sha256": "3e874108914e8f0334da8f12b704261e83755bd25dce6f93e22e8ce22b8dee78",
"raw_report_sha256": "69bcef5ffa58c0f23e5f34641256255bf15f54c836f75fb1aa878b4bd3df1309",
"raw_report_bytes": 267667,
"worker_sha256": "c01cb3ec13cfca722ee768e8350e1399a8b9a4d14a24a35adc3473a5cfc82a73",
"worker": {
"code_query_supported": true,
"context_tokens": 2048,
"device": "cpu",
"dimensions": 768,
"media_supported": true,
"model": "google/embeddinggemma-2",
"ready": true,
"role": "indexing",
"runtime": "b11461-no-logits-v1",
"threads": 1
},
"encoding_ms": 88089.5138,
"media_inference_calls": 10,
"metadata_inference_calls": 16,
"metrics": {
"audio_only": {
"positive_queries": 14,
"top1": 8,
"recall_at_5": 14,
"mrr": 0.7261904761904762,
"no_answer_queries": 4,
"no_answer_returned_any": 4
},
"original": {
"positive_queries": 14,
"top1": 11,
"recall_at_5": 14,
"mrr": 0.869047619047619,
"no_answer_queries": 4,
"no_answer_returned_any": 4
},
"missing_metadata": {
"positive_queries": 14,
"top1": 8,
"recall_at_5": 14,
"mrr": 0.7261904761904762,
"no_answer_queries": 4,
"no_answer_returned_any": 4
},
"misleading_metadata": {
"positive_queries": 14,
"top1": 5,
"recall_at_5": 14,
"mrr": 0.5702380952380952,
"no_answer_queries": 4,
"no_answer_returned_any": 4
}
},
"changes_vs_audio": {
"original": {
"top1_gains_vs_audio": [
"unseen02-en",
"unseen02-de",
"unseen06-en",
"unseen06-de"
],
"top1_losses_vs_audio": [
"unseen01-de"
]
},
"missing_metadata": {
"top1_gains_vs_audio": [],
"top1_losses_vs_audio": []
},
"misleading_metadata": {
"top1_gains_vs_audio": [],
"top1_losses_vs_audio": [
"unseen01-de",
"unseen05-en",
"unseen05-de"
]
}
},
"rows": [
{
"id": "unseen01-en",
"expected": [
"unseen01"
],
"language": "en",
"category": "instrument",
"lanes": {
"audio_only": {
"first_relevant_rank": 1,
"returned_count": 10,
"first_file": "unseen01",
"first_source_kind": "audio",
"first_source_field": null,
"first_score": 0.6707203058852982
},
"original": {
"first_relevant_rank": 1,
"returned_count": 10,
"first_file": "unseen01",
"first_source_kind": "audio",
"first_source_field": null,
"first_score": 0.6707203058852982
},
"missing_metadata": {
"first_relevant_rank": 1,
"returned_count": 10,
"first_file": "unseen01",
"first_source_kind": "audio",
"first_source_field": null,
"first_score": 0.6707203058852982
},
"misleading_metadata": {
"first_relevant_rank": 1,
"returned_count": 10,
"first_file": "unseen01",
"first_source_kind": "audio",
"first_source_field": null,
"first_score": 0.6707203058852982
}
}
},
{
"id": "unseen01-de",
"expected": [
"unseen01"
],
"language": "de",
"category": "instrument",
"lanes": {
"audio_only": {
"first_relevant_rank": 1,
"returned_count": 10,
"first_file": "unseen01",
"first_source_kind": "audio",
"first_source_field": null,
"first_score": 0.6558122105336331
},
"original": {
"first_relevant_rank": 2,
"returned_count": 10,
"first_file": "unseen05",
"first_source_kind": "metadata",
"first_source_field": "title",
"first_score": 0.6588859671020654
},
"missing_metadata": {
"first_relevant_rank": 1,
"returned_count": 10,
"first_file": "unseen01",
"first_source_kind": "audio",
"first_source_field": null,
"first_score": 0.6558122105336331
},
"misleading_metadata": {
"first_relevant_rank": 2,
"returned_count": 10,
"first_file": "unseen04",
"first_source_kind": "metadata",
"first_source_field": "title",
"first_score": 0.6588859671020654
}
}
},
{
"id": "unseen02-en",
"expected": [
"unseen02"
],
"language": "en",
"category": "instrument",
"lanes": {
"audio_only": {
"first_relevant_rank": 2,
"returned_count": 10,
"first_file": "unseen05",
"first_source_kind": "audio",
"first_source_field": null,
"first_score": 0.6690030556732439
},
"original": {
"first_relevant_rank": 1,
"returned_count": 10,
"first_file": "unseen02",
"first_source_kind": "metadata",
"first_source_field": "title",
"first_score": 0.7723038021644638
},
"missing_metadata": {
"first_relevant_rank": 2,
"returned_count": 10,
"first_file": "unseen05",
"first_source_kind": "audio",
"first_source_field": null,
"first_score": 0.6690030556732439
},
"misleading_metadata": {
"first_relevant_rank": 4,
"returned_count": 10,
"first_file": "unseen01",
"first_source_kind": "metadata",
"first_source_field": "title",
"first_score": 0.7723038021644638
}
}
},
{
"id": "unseen02-de",
"expected": [
"unseen02"
],
"language": "de",
"category": "instrument",
"lanes": {
"audio_only": {
"first_relevant_rank": 2,
"returned_count": 10,
"first_file": "unseen05",
"first_source_kind": "audio",
"first_source_field": null,
"first_score": 0.667712688085295
},
"original": {
"first_relevant_rank": 1,
"returned_count": 10,
"first_file": "unseen02",
"first_source_kind": "metadata",
"first_source_field": "title",
"first_score": 0.7157722798891778
},
"missing_metadata": {
"first_relevant_rank": 2,
"returned_count": 10,
"first_file": "unseen05",
"first_source_kind": "audio",
"first_source_field": null,
"first_score": 0.667712688085295
},
"misleading_metadata": {
"first_relevant_rank": 4,
"returned_count": 10,
"first_file": "unseen01",
"first_source_kind": "metadata",
"first_source_field": "title",
"first_score": 0.7157722798891778
}
}
},
{
"id": "unseen03-en",
"expected": [
"unseen03"
],
"language": "en",
"category": "instrument",
"lanes": {
"audio_only": {
"first_relevant_rank": 1,
"returned_count": 10,
"first_file": "unseen03",
"first_source_kind": "audio",
"first_source_field": null,
"first_score": 0.6901545281005906
},
"original": {
"first_relevant_rank": 1,
"returned_count": 10,
"first_file": "unseen03",
"first_source_kind": "audio",
"first_source_field": null,
"first_score": 0.6901545281005906
},
"missing_metadata": {
"first_relevant_rank": 1,
"returned_count": 10,
"first_file": "unseen03",
"first_source_kind": "audio",
"first_source_field": null,
"first_score": 0.6901545281005906
},
"misleading_metadata": {
"first_relevant_rank": 1,
"returned_count": 10,
"first_file": "unseen03",
"first_source_kind": "audio",
"first_source_field": null,
"first_score": 0.6901545281005906
}
}
},
{
"id": "unseen03-de",
"expected": [
"unseen03"
],
"language": "de",
"category": "instrument",
"lanes": {
"audio_only": {
"first_relevant_rank": 1,
"returned_count": 10,
"first_file": "unseen03",
"first_source_kind": "audio",
"first_source_field": null,
"first_score": 0.6759388311775514
},
"original": {
"first_relevant_rank": 1,
"returned_count": 10,
"first_file": "unseen03",
"first_source_kind": "audio",
"first_source_field": null,
"first_score": 0.6759388311775514
},
"missing_metadata": {
"first_relevant_rank": 1,
"returned_count": 10,
"first_file": "unseen03",
"first_source_kind": "audio",
"first_source_field": null,
"first_score": 0.6759388311775514
},
"misleading_metadata": {
"first_relevant_rank": 1,
"returned_count": 10,
"first_file": "unseen03",
"first_source_kind": "audio",
"first_source_field": null,
"first_score": 0.6759388311775514
}
}
},
{
"id": "unseen04-en",
"expected": [
"unseen04"
],
"language": "en",
"category": "instrument",
"lanes": {
"audio_only": {
"first_relevant_rank": 3,
"returned_count": 10,
"first_file": "unseen05",
"first_source_kind": "audio",
"first_source_field": null,
"first_score": 0.7212670979690754
},
"original": {
"first_relevant_rank": 3,
"returned_count": 10,
"first_file": "unseen05",
"first_source_kind": "audio",
"first_source_field": null,
"first_score": 0.7212670979690754
},
"missing_metadata": {
"first_relevant_rank": 3,
"returned_count": 10,
"first_file": "unseen05",
"first_source_kind": "audio",
"first_source_field": null,
"first_score": 0.7212670979690754
},
"misleading_metadata": {
"first_relevant_rank": 4,
"returned_count": 10,
"first_file": "unseen05",
"first_source_kind": "audio",
"first_source_field": null,
"first_score": 0.7212670979690754
}
}
},
{
"id": "unseen04-de",
"expected": [
"unseen04"
],
"language": "de",
"category": "instrument",
"lanes": {
"audio_only": {
"first_relevant_rank": 3,
"returned_count": 10,
"first_file": "unseen05",
"first_source_kind": "audio",
"first_source_field": null,
"first_score": 0.7094035244502205
},
"original": {
"first_relevant_rank": 3,
"returned_count": 10,
"first_file": "unseen05",
"first_source_kind": "audio",
"first_source_field": null,
"first_score": 0.7094035244502205
},
"missing_metadata": {
"first_relevant_rank": 3,
"returned_count": 10,
"first_file": "unseen05",
"first_source_kind": "audio",
"first_source_field": null,
"first_score": 0.7094035244502205
},
"misleading_metadata": {
"first_relevant_rank": 3,
"returned_count": 10,
"first_file": "unseen05",
"first_source_kind": "audio",
"first_source_field": null,
"first_score": 0.7094035244502205
}
}
},
{
"id": "unseen05-en",
"expected": [
"unseen05"
],
"language": "en",
"category": "instrument",
"lanes": {
"audio_only": {
"first_relevant_rank": 1,
"returned_count": 10,
"first_file": "unseen05",
"first_source_kind": "audio",
"first_source_field": null,
"first_score": 0.6986278621913614
},
"original": {
"first_relevant_rank": 1,
"returned_count": 10,
"first_file": "unseen05",
"first_source_kind": "metadata",
"first_source_field": "title",
"first_score": 0.7804412810940734
},
"missing_metadata": {
"first_relevant_rank": 1,
"returned_count": 10,
"first_file": "unseen05",
"first_source_kind": "audio",
"first_source_field": null,
"first_score": 0.6986278621913614
},
"misleading_metadata": {
"first_relevant_rank": 2,
"returned_count": 10,
"first_file": "unseen04",
"first_source_kind": "metadata",
"first_source_field": "title",
"first_score": 0.7804412810940734
}
}
},
{
"id": "unseen05-de",
"expected": [
"unseen05"
],
"language": "de",
"category": "instrument",
"lanes": {
"audio_only": {
"first_relevant_rank": 1,
"returned_count": 10,
"first_file": "unseen05",
"first_source_kind": "audio",
"first_source_field": null,
"first_score": 0.7104005169326468
},
"original": {
"first_relevant_rank": 1,
"returned_count": 10,
"first_file": "unseen05",
"first_source_kind": "metadata",
"first_source_field": "title",
"first_score": 0.7544482777868109
},
"missing_metadata": {
"first_relevant_rank": 1,
"returned_count": 10,
"first_file": "unseen05",
"first_source_kind": "audio",
"first_source_field": null,
"first_score": 0.7104005169326468
},
"misleading_metadata": {
"first_relevant_rank": 2,
"returned_count": 10,
"first_file": "unseen04",
"first_source_kind": "metadata",
"first_source_field": "title",
"first_score": 0.7544482777868109
}
}
},
{
"id": "unseen06-en",
"expected": [
"unseen06"
],
"language": "en",
"category": "instrument",
"lanes": {
"audio_only": {
"first_relevant_rank": 4,
"returned_count": 10,
"first_file": "unseen05",
"first_source_kind": "audio",
"first_source_field": null,
"first_score": 0.7048521660816882
},
"original": {
"first_relevant_rank": 1,
"returned_count": 10,
"first_file": "unseen06",
"first_source_kind": "metadata",
"first_source_field": "title",
"first_score": 0.7621619204850741
},
"missing_metadata": {
"first_relevant_rank": 4,
"returned_count": 10,
"first_file": "unseen05",
"first_source_kind": "audio",
"first_source_field": null,
"first_score": 0.7048521660816882
},
"misleading_metadata": {
"first_relevant_rank": 5,
"returned_count": 10,
"first_file": "unseen05",
"first_source_kind": "metadata",
"first_source_field": "title",
"first_score": 0.7621619204850741
}
}
},
{
"id": "unseen06-de",
"expected": [
"unseen06"
],
"language": "de",
"category": "instrument",
"lanes": {
"audio_only": {
"first_relevant_rank": 4,
"returned_count": 10,
"first_file": "unseen05",
"first_source_kind": "audio",
"first_source_field": null,
"first_score": 0.7064829076815999
},
"original": {
"first_relevant_rank": 1,
"returned_count": 10,
"first_file": "unseen06",
"first_source_kind": "metadata",
"first_source_field": "title",
"first_score": 0.7392143518999531
},
"missing_metadata": {
"first_relevant_rank": 4,
"returned_count": 10,
"first_file": "unseen05",
"first_source_kind": "audio",
"first_source_field": null,
"first_score": 0.7064829076815999
},
"misleading_metadata": {
"first_relevant_rank": 5,
"returned_count": 10,
"first_file": "unseen05",
"first_source_kind": "metadata",
"first_source_field": "title",
"first_score": 0.7392143518999531
}
}
},
{
"id": "sax-broad-en",
"expected": [
"unseen05",
"unseen06"
],
"language": "en",
"category": "related-positive",
"lanes": {
"audio_only": {
"first_relevant_rank": 1,
"returned_count": 10,
"first_file": "unseen05",
"first_source_kind": "audio",
"first_source_field": null,
"first_score": 0.7506144568545938
},
"original": {
"first_relevant_rank": 1,
"returned_count": 10,
"first_file": "unseen06",
"first_source_kind": "metadata",
"first_source_field": "title",
"first_score": 0.7558704108318935
},
"missing_metadata": {
"first_relevant_rank": 1,
"returned_count": 10,
"first_file": "unseen05",
"first_source_kind": "audio",
"first_source_field": null,
"first_score": 0.7506144568545938
},
"misleading_metadata": {
"first_relevant_rank": 1,
"returned_count": 10,
"first_file": "unseen05",
"first_source_kind": "metadata",
"first_source_field": "title",
"first_score": 0.7558704108318935
}
}
},
{
"id": "sax-broad-de",
"expected": [
"unseen05",
"unseen06"
],
"language": "de",
"category": "related-positive",
"lanes": {
"audio_only": {
"first_relevant_rank": 1,
"returned_count": 10,
"first_file": "unseen05",
"first_source_kind": "audio",
"first_source_field": null,
"first_score": 0.7499595252586609
},
"original": {
"first_relevant_rank": 1,
"returned_count": 10,
"first_file": "unseen06",
"first_source_kind": "metadata",
"first_source_field": "title",
"first_score": 0.762964312138735
},
"missing_metadata": {
"first_relevant_rank": 1,
"returned_count": 10,
"first_file": "unseen05",
"first_source_kind": "audio",
"first_source_field": null,
"first_score": 0.7499595252586609
},
"misleading_metadata": {
"first_relevant_rank": 1,
"returned_count": 10,
"first_file": "unseen05",
"first_source_kind": "metadata",
"first_source_field": "title",
"first_score": 0.762964312138735
}
}
},
{
"id": "absent-bagpipe-en",
"expected": [],
"language": "en",
"category": "absent-instrument",
"lanes": {
"audio_only": {
"first_relevant_rank": null,
"returned_count": 10,
"first_file": "unseen05",
"first_source_kind": "audio",
"first_source_field": null,
"first_score": 0.6899061391978294
},
"original": {
"first_relevant_rank": null,
"returned_count": 10,
"first_file": "unseen05",
"first_source_kind": "audio",
"first_source_field": null,
"first_score": 0.6899061391978294
},
"missing_metadata": {
"first_relevant_rank": null,
"returned_count": 10,
"first_file": "unseen05",
"first_source_kind": "audio",
"first_source_field": null,
"first_score": 0.6899061391978294
},
"misleading_metadata": {
"first_relevant_rank": null,
"returned_count": 10,
"first_file": "unseen05",
"first_source_kind": "audio",
"first_source_field": null,
"first_score": 0.6899061391978294
}
}
},
{
"id": "absent-bagpipe-de",
"expected": [],
"language": "de",
"category": "absent-instrument",
"lanes": {
"audio_only": {
"first_relevant_rank": null,
"returned_count": 10,
"first_file": "unseen05",
"first_source_kind": "audio",
"first_source_field": null,
"first_score": 0.6517667406795858
},
"original": {
"first_relevant_rank": null,
"returned_count": 10,
"first_file": "unseen05",
"first_source_kind": "audio",
"first_source_field": null,
"first_score": 0.6517667406795858
},
"missing_metadata": {
"first_relevant_rank": null,
"returned_count": 10,
"first_file": "unseen05",
"first_source_kind": "audio",
"first_source_field": null,
"first_score": 0.6517667406795858
},
"misleading_metadata": {
"first_relevant_rank": null,
"returned_count": 10,
"first_file": "unseen05",
"first_source_kind": "audio",
"first_source_field": null,
"first_score": 0.6517667406795858
}
}
},
{
"id": "absent-artist-en",
"expected": [],
"language": "en",
"category": "absent-identity",
"lanes": {
"audio_only": {
"first_relevant_rank": null,
"returned_count": 10,
"first_file": "sound02",
"first_source_kind": "audio",
"first_source_field": null,
"first_score": 0.5977197385044184
},
"original": {
"first_relevant_rank": null,
"returned_count": 10,
"first_file": "unseen05",
"first_source_kind": "metadata",
"first_source_field": "title",
"first_score": 0.5983290055413841
},
"missing_metadata": {
"first_relevant_rank": null,
"returned_count": 10,
"first_file": "sound02",
"first_source_kind": "audio",
"first_source_field": null,
"first_score": 0.5977197385044184
},
"misleading_metadata": {
"first_relevant_rank": null,
"returned_count": 10,
"first_file": "unseen04",
"first_source_kind": "metadata",
"first_source_field": "title",
"first_score": 0.5983290055413841
}
}
},
{
"id": "absent-content-en",
"expected": [],
"language": "en",
"category": "absent-content",
"lanes": {
"audio_only": {
"first_relevant_rank": null,
"returned_count": 10,
"first_file": "unseen01",
"first_source_kind": "audio",
"first_source_field": null,
"first_score": 0.5870996954366186
},
"original": {
"first_relevant_rank": null,
"returned_count": 10,
"first_file": "unseen01",
"first_source_kind": "audio",
"first_source_field": null,
"first_score": 0.5870996954366186
},
"missing_metadata": {
"first_relevant_rank": null,
"returned_count": 10,
"first_file": "unseen01",
"first_source_kind": "audio",
"first_source_field": null,
"first_score": 0.5870996954366186
},
"misleading_metadata": {
"first_relevant_rank": null,
"returned_count": 10,
"first_file": "unseen01",
"first_source_kind": "audio",
"first_source_field": null,
"first_score": 0.5870996954366186
}
}
}
],
"environment": {
"os": "Windows11 Pro10.0.26300",
"logical_cpus": 8,
"visible_memory_kib": 16708012,
"processor": "AMD Ryzen7 7800X3D",
"shared_vm": true,
"existing_app_query_worker": "Warmed CPU session remained idle; no root model requests during reserved slot"
},
"production_policy_changed": false,
"deployment_conclusion": "Original metadata improves aggregate retrieval but introduces a previously-correct German guitar failure; misleading metadata reduces top1 and no-answer remains unsolved. Unconditional max policy not validated for deployment. No post-score policy tuning."
}
tools/rerank-eval/README.md
+44
View File
@@ -0,0 +1,44 @@
# Local cross-encoder diagnostic
This standalone research tool never modifies production search or its Google
EmbeddingGemma 2 model. Do not confuse the cross-encoder's name with the historical
MiniLM embedding experiment. Publisher weights stay under ignored `.searchi` data.
Pinned publisher revision, asset bytes/SHA-256 and licence provenance are in
`model-manifest.json`; the pre-inference contract is in `compatibility-plan.json`.
The existing CPU Python environment can execute it without installing Torch or
changing production dependencies:
```powershell
python tools/rerank-eval/download.py
$env:SEARCHI_FORCE_CPU = '1'
& tools/semantic-proof/.venv/Scripts/python.exe tools/rerank-eval/compatibility.py --output .searchi/rerank-eval/compatibility-v1.json
```
The runner verifies all assets, rejects changed frozen inputs, enables only CPU
Execution Provider with one thread, bounds five pairs to 256 tokens including
special tokens and at most 64 query tokens, checks finite one-logit-per-pair
outputs, and records serial/batch parity and twenty warm timing samples. It creates
new reports atomically and refuses overwriting prior evidence. A failed parity
contract writes the observed report then exits nonzero. Never weaken its frozen
tolerance after observing output.
The first execution failed serial/batch logit parity; see
[recorded result](../../docs/LOCAL-RERANKING.md). It is not a product integration,
relevance calibration or full search performance benchmark. Scores are not
probabilities that the requested artifact exists. Any further experimental input
or scoring policy needs a separate version and output path.
The separate unquantized follow-up preserves the first failed evidence:
```powershell
python tools/rerank-eval/download.py --reference
$env:SEARCHI_FORCE_CPU = '1'
& tools/semantic-proof/.venv/Scripts/python.exe tools/rerank-eval/reference.py --output .searchi/rerank-eval/reference-v1.json
```
`reference-plan.json` retains five short pairs and adds a 256-token budget case.
F32 parity passed; quantized score differences persisted. Mixed-length padded
batching was slower. Neither result establishes general absence calibration or
sufficient default-search latency. Detailed costs and possible VM contention are
recorded in the linked report documentation.
tools/rerank-eval/compatibility-plan.json
+38
View File
@@ -0,0 +1,38 @@
{
"version": 1,
"status": "pre-inference frozen compatibility/cost contract, not relevance calibration",
"cpu_provider": "CPUExecutionProvider",
"intra_op_threads": 1,
"inter_op_threads": 1,
"parallel_execution": false,
"max_pairs": 5,
"max_pair_tokens": 256,
"max_query_tokens": 64,
"warm_iterations": 20,
"parity_absolute_tolerance": 0.001,
"parity_relative_tolerance": 0.001,
"added_warm_p95_target_ms": 200,
"no_suppression": true,
"pairs": [
[
"How do we cancel the supplier partnership?",
"Either party may terminate the supplier contract by giving sixty days written notice."
],
[
"Wie beenden wir die Zusammenarbeit mit dem Lieferanten?",
"Jede Partei kann die Zusammenarbeit mit dem Lieferanten mit einer Frist von sechzig Tagen kündigen."
],
[
"How do we cancel the supplier partnership?",
"Tomatoes require sunshine and consistent watering."
],
[
"Wie beenden wir die Zusammenarbeit mit dem Lieferanten?",
"Der Garten benötigt Sonnenlicht und regelmäßige Bewässerung."
],
[
"Where does the code implement retry backoff?",
"fn retry_delay(attempt: u32) -> u64 { 100 * (1 << attempt.min(6)) }"
]
]
}
tools/rerank-eval/compatibility.py
+131
View File
@@ -0,0 +1,131 @@
"""Owned CPU-only cross-encoder compatibility/cost diagnostic; never alters search."""
import argparse
import hashlib
import json
import math
import os
from pathlib import Path
import platform
import statistics
import tempfile
import time
import numpy as np
import onnxruntime as ort
from tokenizers import Tokenizer
from download import DESTINATION, digest
EXPECTED_PLAN = "02569e771ab74ff593842ff26d7eb42d1b0509bef09a1867cf0f1be6b1dc5a5a"
EXPECTED_MANIFEST = "92ec98e8ccb5f8e8e08fbf46c7a4eb1407479b4dd0fafaabf6dc7cfc7aae2da7"
def checked_json(path, expected):
data = path.read_bytes()
if hashlib.sha256(data).hexdigest() != expected:
raise ValueError("Frozen diagnostic inputs changed")
return json.loads(data)
def main():
parser = argparse.ArgumentParser()
parser.add_argument("--output", type=Path, required=True)
args = parser.parse_args()
owned_root = DESTINATION.parents[1].resolve()
output = args.output.resolve()
if not output.is_relative_to(owned_root) or output.exists():
raise ValueError("Output must be new and under owned .searchi data")
if os.environ.get("SEARCHI_FORCE_CPU") != "1":
raise ValueError("Windows VM diagnostic requires explicit SEARCHI_FORCE_CPU=1")
plan = checked_json(Path(__file__).with_name("compatibility-plan.json"), EXPECTED_PLAN)
manifest = checked_json(Path(__file__).with_name("model-manifest.json"), EXPECTED_MANIFEST)
start = time.perf_counter()
for item in manifest["files"]:
path = DESTINATION / item["path"]
if path.stat().st_size != item["bytes"] or digest(path) != item["sha256"]:
raise ValueError("Pinned diagnostic asset mismatch")
verification_ms = (time.perf_counter() - start) * 1000
start = time.perf_counter()
tokenizer = Tokenizer.from_file(str(DESTINATION / "tokenizer.json"))
special = tokenizer.num_special_tokens_to_add(True)
encodings = []
for query, passage in plan["pairs"]:
query_encoding = tokenizer.encode(query, add_special_tokens=False)
query_encoding.truncate(plan["max_query_tokens"])
passage_encoding = tokenizer.encode(passage, add_special_tokens=False)
passage_encoding.truncate(plan["max_pair_tokens"] - len(query_encoding.ids) - special)
encoding = tokenizer.post_process(query_encoding, passage_encoding, add_special_tokens=True)
if len(encoding.ids) > plan["max_pair_tokens"]:
raise ValueError("Pair token budget exceeded")
encodings.append(encoding)
tokenization_ms = (time.perf_counter() - start) * 1000
options = ort.SessionOptions()
options.intra_op_num_threads = 1
options.inter_op_num_threads = 1
options.execution_mode = ort.ExecutionMode.ORT_SEQUENTIAL
start = time.perf_counter()
session = ort.InferenceSession(str(DESTINATION / "onnx/model_quint8_avx2.onnx"), sess_options=options, providers=["CPUExecutionProvider"])
startup_ms = (time.perf_counter() - start) * 1000
if session.get_providers() != ["CPUExecutionProvider"]:
raise ValueError("Unexpected execution provider")
names = [item.name for item in session.get_inputs()]
if not set(names) <= {"input_ids", "attention_mask", "token_type_ids"}:
raise ValueError("Unexpected model input contract")
def score(items):
width = max(len(item.ids) for item in items)
ids = np.full((len(items), width), 1, dtype=np.int64)
mask = np.zeros_like(ids)
types = np.zeros_like(ids)
for index, item in enumerate(items):
count = len(item.ids)
ids[index, :count] = item.ids
mask[index, :count] = item.attention_mask
types[index, :count] = item.type_ids
values = {"input_ids": ids, "attention_mask": mask, "token_type_ids": types}
logits = session.run(None, {name: values[name] for name in names})[0]
if logits.shape != (len(items), 1) or not np.isfinite(logits).all():
raise ValueError("Invalid classifier output")
return logits[:, 0]
start = time.perf_counter()
serial = np.asarray([float(score([item])[0]) for item in encodings])
first_serial_ms = (time.perf_counter() - start) * 1000
start = time.perf_counter()
batch = score(encodings)
first_batch_ms = (time.perf_counter() - start) * 1000
parity = bool(np.allclose(serial, batch, atol=plan["parity_absolute_tolerance"], rtol=plan["parity_relative_tolerance"]))
serial_timings = []
batch_timings = []
for _ in range(plan["warm_iterations"]):
start = time.perf_counter()
for item in encodings:
score([item])
serial_timings.append((time.perf_counter() - start) * 1000)
start = time.perf_counter()
score(encodings)
batch_timings.append((time.perf_counter() - start) * 1000)
def timing(values):
return {"median_ms": statistics.median(values), "p95_ms": sorted(values)[math.ceil(len(values) * .95) - 1], "samples_ms": values}
report = {"complete": True, "plan_sha256": EXPECTED_PLAN, "manifest_sha256": EXPECTED_MANIFEST,
"platform": platform.platform(), "python": platform.python_version(), "onnxruntime": ort.__version__,
"numpy": np.__version__, "providers": session.get_providers(), "cpu_threads": 1,
"asset_verification_ms": verification_ms, "tokenization_ms": tokenization_ms, "session_startup_ms": startup_ms,
"pair_tokens": [len(item.ids) for item in encodings], "first_serial_ms": first_serial_ms, "first_batch_ms": first_batch_ms,
"serial_logits": serial.tolist(), "batch_logits": batch.tolist(), "parity_passed": parity,
"max_absolute_difference": float(np.max(np.abs(serial - batch))),
"warm_serial": timing(serial_timings), "warm_batch": timing(batch_timings),
"limitations": "Five synthetic short pairs; classifier compatibility/cost only. No search pipeline or absence calibration. No confidence probability or suppression."}
output.parent.mkdir(parents=True, exist_ok=True)
with tempfile.NamedTemporaryFile(mode="w", encoding="utf-8", dir=output.parent, delete=False) as handle:
temporary = Path(handle.name)
json.dump(report, handle, indent=2)
handle.write("\n")
try:
os.link(temporary, output)
finally:
temporary.unlink(missing_ok=True)
print(json.dumps({"output": str(output), "parity_passed": parity, "max_absolute_difference": report["max_absolute_difference"], "warm_serial": report["warm_serial"]["p95_ms"], "warm_batch": report["warm_batch"]["p95_ms"]}), flush=True)
if not parity:
raise SystemExit("Serial/batch parity failed; do not deploy")
if __name__ == "__main__":
main()
tools/rerank-eval/download.py
+53
View File
@@ -0,0 +1,53 @@
"""Download only pinned diagnostic assets into owned ignored data."""
import argparse
import hashlib
import json
from pathlib import Path
import time
import urllib.request
import uuid
ROOT = Path(__file__).resolve().parents[2]
DESTINATION = ROOT / ".searchi" / "rerank-eval" / "model"
def digest(path):
checksum = hashlib.sha256()
with path.open("rb") as handle:
for chunk in iter(lambda: handle.read(65536), b""):
checksum.update(chunk)
return checksum.hexdigest()
def main():
parser=argparse.ArgumentParser()
parser.add_argument("--reference",action="store_true")
args=parser.parse_args()
name="reference-manifest.json" if args.reference else "model-manifest.json"
manifest = json.loads(Path(__file__).with_name(name).read_text(encoding="utf-8"))
for item in manifest["files"]:
destination = DESTINATION / item["path"]
if destination.is_file() and destination.stat().st_size == item["bytes"] and digest(destination) == item["sha256"]:
print("verified cache", item["path"], flush=True)
continue
destination.parent.mkdir(parents=True, exist_ok=True)
temporary = destination.with_name(destination.name + "." + uuid.uuid4().hex + ".partial")
total = 0
start = time.monotonic()
try:
with urllib.request.urlopen(item["url"], timeout=20) as response, temporary.open("xb") as output:
while chunk := response.read(65536):
total += len(chunk)
if total > item["bytes"] or time.monotonic() - start > 180:
raise ValueError("Download exceeded size/deadline")
output.write(chunk)
if total != item["bytes"] or digest(temporary) != item["sha256"]:
raise ValueError("Pinned asset hash/size mismatch")
temporary.replace(destination)
print("verified download", item["path"], total, flush=True)
finally:
temporary.unlink(missing_ok=True)
if __name__ == "__main__":
main()
tools/rerank-eval/model-manifest.json
+50
View File
@@ -0,0 +1,50 @@
{
"model_id": "cross-encoder/mmarco-mMiniLMv2-L12-H384-v1",
"revision": "1427fd652930e4ba29e8149678df786c240d8825",
"purpose": "diagnostic text reranker; Google EmbeddingGemma2 remains retrieval",
"license": "Apache-2.0 per pinned publisher card; text from Apache",
"files": [
{
"path": "onnx/model_quint8_avx2.onnx",
"bytes": 118620016,
"sha256": "6c2513767fb63d008a4377bef7a7a3555433d9436342bb53e35a3a72ffc52d4b",
"url": "https://huggingface.co/cross-encoder/mmarco-mMiniLMv2-L12-H384-v1/resolve/1427fd652930e4ba29e8149678df786c240d8825/onnx/model_quint8_avx2.onnx"
},
{
"path": "tokenizer.json",
"bytes": 17082660,
"sha256": "62c24cdc13d4c9952d63718d6c9fa4c287974249e16b7ade6d5a85e7bbb75626",
"url": "https://huggingface.co/cross-encoder/mmarco-mMiniLMv2-L12-H384-v1/resolve/1427fd652930e4ba29e8149678df786c240d8825/tokenizer.json"
},
{
"path": "config.json",
"bytes": 891,
"sha256": "cc2cfe51aa3fd759d21d21acf5dfd6994aa67a3c9210636d22e143699d336c77",
"url": "https://huggingface.co/cross-encoder/mmarco-mMiniLMv2-L12-H384-v1/resolve/1427fd652930e4ba29e8149678df786c240d8825/config.json"
},
{
"path": "tokenizer_config.json",
"bytes": 435,
"sha256": "e7fbfbfa6347b4e414c1cee50d142e2c2f9a895dad68b068ae83a8b564c3837e",
"url": "https://huggingface.co/cross-encoder/mmarco-mMiniLMv2-L12-H384-v1/resolve/1427fd652930e4ba29e8149678df786c240d8825/tokenizer_config.json"
},
{
"path": "special_tokens_map.json",
"bytes": 239,
"sha256": "378eb3bf733eb16e65792d7e3fda5b8a4631387ca04d2015199c4d4f22ae554d",
"url": "https://huggingface.co/cross-encoder/mmarco-mMiniLMv2-L12-H384-v1/resolve/1427fd652930e4ba29e8149678df786c240d8825/special_tokens_map.json"
},
{
"path": "README.md",
"bytes": 2278,
"sha256": "474736a65d6393a060119a8dc304563af67af4d8d86ccfee4a05dd0df107fc11",
"url": "https://huggingface.co/cross-encoder/mmarco-mMiniLMv2-L12-H384-v1/resolve/1427fd652930e4ba29e8149678df786c240d8825/README.md"
},
{
"path": "LICENSE-APACHE-2.0.txt",
"bytes": 11358,
"sha256": "cfc7749b96f63bd31c3c42b5c471bf756814053e847c10f3eb003417bc523d30",
"url": "https://www.apache.org/licenses/LICENSE-2.0.txt"
}
]
}
tools/rerank-eval/reference-manifest.json
+56
View File
@@ -0,0 +1,56 @@
{
"model_id": "cross-encoder/mmarco-mMiniLMv2-L12-H384-v1",
"revision": "1427fd652930e4ba29e8149678df786c240d8825",
"purpose": "diagnostic text reranker; Google EmbeddingGemma2 remains retrieval",
"license": "Apache-2.0 per pinned publisher card; text from Apache",
"files": [
{
"path": "onnx/model_quint8_avx2.onnx",
"bytes": 118620016,
"sha256": "6c2513767fb63d008a4377bef7a7a3555433d9436342bb53e35a3a72ffc52d4b",
"url": "https://huggingface.co/cross-encoder/mmarco-mMiniLMv2-L12-H384-v1/resolve/1427fd652930e4ba29e8149678df786c240d8825/onnx/model_quint8_avx2.onnx"
},
{
"path": "tokenizer.json",
"bytes": 17082660,
"sha256": "62c24cdc13d4c9952d63718d6c9fa4c287974249e16b7ade6d5a85e7bbb75626",
"url": "https://huggingface.co/cross-encoder/mmarco-mMiniLMv2-L12-H384-v1/resolve/1427fd652930e4ba29e8149678df786c240d8825/tokenizer.json"
},
{
"path": "config.json",
"bytes": 891,
"sha256": "cc2cfe51aa3fd759d21d21acf5dfd6994aa67a3c9210636d22e143699d336c77",
"url": "https://huggingface.co/cross-encoder/mmarco-mMiniLMv2-L12-H384-v1/resolve/1427fd652930e4ba29e8149678df786c240d8825/config.json"
},
{
"path": "tokenizer_config.json",
"bytes": 435,
"sha256": "e7fbfbfa6347b4e414c1cee50d142e2c2f9a895dad68b068ae83a8b564c3837e",
"url": "https://huggingface.co/cross-encoder/mmarco-mMiniLMv2-L12-H384-v1/resolve/1427fd652930e4ba29e8149678df786c240d8825/tokenizer_config.json"
},
{
"path": "special_tokens_map.json",
"bytes": 239,
"sha256": "378eb3bf733eb16e65792d7e3fda5b8a4631387ca04d2015199c4d4f22ae554d",
"url": "https://huggingface.co/cross-encoder/mmarco-mMiniLMv2-L12-H384-v1/resolve/1427fd652930e4ba29e8149678df786c240d8825/special_tokens_map.json"
},
{
"path": "README.md",
"bytes": 2278,
"sha256": "474736a65d6393a060119a8dc304563af67af4d8d86ccfee4a05dd0df107fc11",
"url": "https://huggingface.co/cross-encoder/mmarco-mMiniLMv2-L12-H384-v1/resolve/1427fd652930e4ba29e8149678df786c240d8825/README.md"
},
{
"path": "LICENSE-APACHE-2.0.txt",
"bytes": 11358,
"sha256": "cfc7749b96f63bd31c3c42b5c471bf756814053e847c10f3eb003417bc523d30",
"url": "https://www.apache.org/licenses/LICENSE-2.0.txt"
},
{
"path": "onnx/model.onnx",
"bytes": 470883696,
"sha256": "3e9a03ed1e966f7c5288dd4230e3d6a9bf5e3a170a06f1f4241c5bca12c6487c",
"url": "https://huggingface.co/cross-encoder/mmarco-mMiniLMv2-L12-H384-v1/resolve/1427fd652930e4ba29e8149678df786c240d8825/onnx/model.onnx"
}
]
}
tools/rerank-eval/reference-plan.json
+44
View File
@@ -0,0 +1,44 @@
{
"version": 2,
"status": "pre-inference F32 reference parity and 256-token cost contract; original INT8 failure unchanged",
"cpu_provider": "CPUExecutionProvider",
"intra_op_threads": 1,
"inter_op_threads": 1,
"parallel_execution": false,
"max_pairs": 6,
"max_pair_tokens": 256,
"max_query_tokens": 64,
"warm_iterations": 20,
"parity_absolute_tolerance": 0.001,
"parity_relative_tolerance": 0.001,
"added_warm_p95_target_ms": 200,
"no_suppression": true,
"pairs": [
[
"How do we cancel the supplier partnership?",
"Either party may terminate the supplier contract by giving sixty days written notice."
],
[
"Wie beenden wir die Zusammenarbeit mit dem Lieferanten?",
"Jede Partei kann die Zusammenarbeit mit dem Lieferanten mit einer Frist von sechzig Tagen kündigen."
],
[
"How do we cancel the supplier partnership?",
"Tomatoes require sunshine and consistent watering."
],
[
"Wie beenden wir die Zusammenarbeit mit dem Lieferanten?",
"Der Garten benötigt Sonnenlicht und regelmäßige Bewässerung."
],
[
"Where does the code implement retry backoff?",
"fn retry_delay(attempt: u32) -> u64 { 100 * (1 << attempt.min(6)) }"
],
[
"supplier partnership cancellation notice supplier partnership cancellation notice supplier partnership cancellation notice supplier partnership cancellation notice supplier partnership cancellation notice supplier partnership cancellation notice supplier partnership cancellation notice supplier partnership cancellation notice supplier partnership cancellation notice supplier partnership cancellation notice supplier partnership cancellation notice supplier partnership cancellation notice supplier partnership cancellation notice supplier partnership cancellation notice supplier partnership cancellation notice supplier partnership cancellation notice supplier partnership cancellation notice supplier partnership cancellation notice supplier partnership cancellation notice supplier partnership cancellation notice supplier partnership cancellation notice supplier partnership cancellation notice supplier partnership cancellation notice supplier partnership cancellation notice supplier partnership cancellation notice supplier partnership cancellation notice supplier partnership cancellation notice supplier partnership cancellation notice supplier partnership cancellation notice supplier partnership cancellation notice ",
"Either party may terminate the supplier partnership with sixty days written notice. Either party may terminate the supplier partnership with sixty days written notice. Either party may terminate the supplier partnership with sixty days written notice. Either party may terminate the supplier partnership with sixty days written notice. Either party may terminate the supplier partnership with sixty days written notice. Either party may terminate the supplier partnership with sixty days written notice. Either party may terminate the supplier partnership with sixty days written notice. Either party may terminate the supplier partnership with sixty days written notice. Either party may terminate the supplier partnership with sixty days written notice. Either party may terminate the supplier partnership with sixty days written notice. Either party may terminate the supplier partnership with sixty days written notice. Either party may terminate the supplier partnership with sixty days written notice. Either party may terminate the supplier partnership with sixty days written notice. Either party may terminate the supplier partnership with sixty days written notice. Either party may terminate the supplier partnership with sixty days written notice. Either party may terminate the supplier partnership with sixty days written notice. Either party may terminate the supplier partnership with sixty days written notice. Either party may terminate the supplier partnership with sixty days written notice. Either party may terminate the supplier partnership with sixty days written notice. Either party may terminate the supplier partnership with sixty days written notice. Either party may terminate the supplier partnership with sixty days written notice. Either party may terminate the supplier partnership with sixty days written notice. Either party may terminate the supplier partnership with sixty days written notice. Either party may terminate the supplier partnership with sixty days written notice. Either party may terminate the supplier partnership with sixty days written notice. Either party may terminate the supplier partnership with sixty days written notice. Either party may terminate the supplier partnership with sixty days written notice. Either party may terminate the supplier partnership with sixty days written notice. Either party may terminate the supplier partnership with sixty days written notice. Either party may terminate the supplier partnership with sixty days written notice. Either party may terminate the supplier partnership with sixty days written notice. Either party may terminate the supplier partnership with sixty days written notice. Either party may terminate the supplier partnership with sixty days written notice. Either party may terminate the supplier partnership with sixty days written notice. Either party may terminate the supplier partnership with sixty days written notice. Either party may terminate the supplier partnership with sixty days written notice. Either party may terminate the supplier partnership with sixty days written notice. Either party may terminate the supplier partnership with sixty days written notice. Either party may terminate the supplier partnership with sixty days written notice. Either party may terminate the supplier partnership with sixty days written notice. Either party may terminate the supplier partnership with sixty days written notice. Either party may terminate the supplier partnership with sixty days written notice. Either party may terminate the supplier partnership with sixty days written notice. Either party may terminate the supplier partnership with sixty days written notice. Either party may terminate the supplier partnership with sixty days written notice. Either party may terminate the supplier partnership with sixty days written notice. Either party may terminate the supplier partnership with sixty days written notice. Either party may terminate the supplier partnership with sixty days written notice. Either party may terminate the supplier partnership with sixty days written notice. Either party may terminate the supplier partnership with sixty days written notice. Either party may terminate the supplier partnership with sixty days written notice. Either party may terminate the supplier partnership with sixty days written notice. Either party may terminate the supplier partnership with sixty days written notice. Either party may terminate the supplier partnership with sixty days written notice. Either party may terminate the supplier partnership with sixty days written notice. Either party may terminate the supplier partnership with sixty days written notice. Either party may terminate the supplier partnership with sixty days written notice. Either party may terminate the supplier partnership with sixty days written notice. Either party may terminate the supplier partnership with sixty days written notice. Either party may terminate the supplLine truncated
]
],
"reference_model": "onnx/model.onnx",
"comparison_model": "onnx/model_quint8_avx2.onnx"
}
tools/rerank-eval/reference.py
+143
View File
@@ -0,0 +1,143 @@
"""Owned CPU-only cross-encoder compatibility/cost diagnostic; never alters search."""
import argparse
import hashlib
import json
import math
import os
from pathlib import Path
import platform
import statistics
import tempfile
import time
import numpy as np
import onnxruntime as ort
from tokenizers import Tokenizer
from download import DESTINATION, digest
EXPECTED_PLAN = "174867d4212b222e5302b1b16832b371c6bd38ec991f4145c952ed35446341f1"
EXPECTED_MANIFEST = "3c252dfc10e03ebba59a7ff0e33e6e2183a26c71862c52f0700f27ba659142be"
def checked_json(path, expected):
data = path.read_bytes()
if hashlib.sha256(data).hexdigest() != expected:
raise ValueError("Frozen diagnostic inputs changed")
return json.loads(data)
def main():
parser = argparse.ArgumentParser()
parser.add_argument("--output", type=Path, required=True)
args = parser.parse_args()
owned_root = DESTINATION.parents[1].resolve()
output = args.output.resolve()
if not output.is_relative_to(owned_root) or output.exists():
raise ValueError("Output must be new and under owned .searchi data")
if os.environ.get("SEARCHI_FORCE_CPU") != "1":
raise ValueError("Windows VM diagnostic requires explicit SEARCHI_FORCE_CPU=1")
plan = checked_json(Path(__file__).with_name("reference-plan.json"), EXPECTED_PLAN)
manifest = checked_json(Path(__file__).with_name("reference-manifest.json"), EXPECTED_MANIFEST)
start = time.perf_counter()
for item in manifest["files"]:
path = DESTINATION / item["path"]
if path.stat().st_size != item["bytes"] or digest(path) != item["sha256"]:
raise ValueError("Pinned diagnostic asset mismatch")
verification_ms = (time.perf_counter() - start) * 1000
start = time.perf_counter()
tokenizer = Tokenizer.from_file(str(DESTINATION / "tokenizer.json"))
special = tokenizer.num_special_tokens_to_add(True)
encodings = []
for query, passage in plan["pairs"]:
query_encoding = tokenizer.encode(query, add_special_tokens=False)
query_encoding.truncate(plan["max_query_tokens"])
passage_encoding = tokenizer.encode(passage, add_special_tokens=False)
passage_encoding.truncate(plan["max_pair_tokens"] - len(query_encoding.ids) - special)
encoding = tokenizer.post_process(query_encoding, passage_encoding, add_special_tokens=True)
if len(encoding.ids) > plan["max_pair_tokens"]:
raise ValueError("Pair token budget exceeded")
encodings.append(encoding)
tokenization_ms = (time.perf_counter() - start) * 1000
options = ort.SessionOptions()
options.intra_op_num_threads = 1
options.inter_op_num_threads = 1
options.execution_mode = ort.ExecutionMode.ORT_SEQUENTIAL
start = time.perf_counter()
session = ort.InferenceSession(str(DESTINATION / plan["reference_model"]), sess_options=options, providers=["CPUExecutionProvider"])
startup_ms = (time.perf_counter() - start) * 1000
if session.get_providers() != ["CPUExecutionProvider"]:
raise ValueError("Unexpected execution provider")
names = [item.name for item in session.get_inputs()]
if not set(names) <= {"input_ids", "attention_mask", "token_type_ids"}:
raise ValueError("Unexpected model input contract")
def score(items):
width = max(len(item.ids) for item in items)
ids = np.full((len(items), width), 1, dtype=np.int64)
mask = np.zeros_like(ids)
types = np.zeros_like(ids)
for index, item in enumerate(items):
count = len(item.ids)
ids[index, :count] = item.ids
mask[index, :count] = item.attention_mask
types[index, :count] = item.type_ids
values = {"input_ids": ids, "attention_mask": mask, "token_type_ids": types}
logits = session.run(None, {name: values[name] for name in names})[0]
if logits.shape != (len(items), 1) or not np.isfinite(logits).all():
raise ValueError("Invalid classifier output")
return logits[:, 0]
start = time.perf_counter()
serial = np.asarray([float(score([item])[0]) for item in encodings])
first_serial_ms = (time.perf_counter() - start) * 1000
start = time.perf_counter()
batch = score(encodings)
first_batch_ms = (time.perf_counter() - start) * 1000
parity = bool(np.allclose(serial, batch, atol=plan["parity_absolute_tolerance"], rtol=plan["parity_relative_tolerance"]))
serial_timings = []
batch_timings = []
for _ in range(plan["warm_iterations"]):
start = time.perf_counter()
for item in encodings:
score([item])
serial_timings.append((time.perf_counter() - start) * 1000)
start = time.perf_counter()
score(encodings)
batch_timings.append((time.perf_counter() - start) * 1000)
reference_session = session
session = ort.InferenceSession(str(DESTINATION / plan["comparison_model"]), sess_options=options, providers=["CPUExecutionProvider"])
if session.get_providers() != ["CPUExecutionProvider"]:
raise ValueError("Unexpected comparison provider")
quantized_serial = np.asarray([float(score([item])[0]) for item in encodings])
quantized_batch = score(encodings)
session = reference_session
def timing(values):
return {"median_ms": statistics.median(values), "p95_ms": sorted(values)[math.ceil(len(values) * .95) - 1], "samples_ms": values}
report = {"complete": True, "plan_sha256": EXPECTED_PLAN, "manifest_sha256": EXPECTED_MANIFEST,
"platform": platform.platform(), "python": platform.python_version(), "onnxruntime": ort.__version__,
"numpy": np.__version__, "providers": session.get_providers(), "cpu_threads": 1,
"asset_verification_ms": verification_ms, "tokenization_ms": tokenization_ms, "session_startup_ms": startup_ms,
"pair_tokens": [len(item.ids) for item in encodings], "first_serial_ms": first_serial_ms, "first_batch_ms": first_batch_ms,
"reference_model": plan["reference_model"], "comparison_model": plan["comparison_model"],
"serial_logits": serial.tolist(), "batch_logits": batch.tolist(), "parity_passed": parity,
"quantized_serial_logits": quantized_serial.tolist(), "quantized_batch_logits": quantized_batch.tolist(),
"quantized_vs_reference_serial_max_abs": float(np.max(np.abs(serial-quantized_serial))),
"quantized_batch_max_abs": float(np.max(np.abs(quantized_serial-quantized_batch))),
"five_short_pairs_ordering_agrees": bool(np.array_equal(np.argsort(serial[:5]),np.argsort(quantized_serial[:5]))),
"max_absolute_difference": float(np.max(np.abs(serial - batch))),
"warm_serial": timing(serial_timings), "warm_batch": timing(batch_timings),
"limitations": "Five synthetic short pairs plus one 256-token cost pair; classifier compatibility/cost only. No search pipeline or absence calibration. No confidence probability or suppression."}
output.parent.mkdir(parents=True, exist_ok=True)
with tempfile.NamedTemporaryFile(mode="w", encoding="utf-8", dir=output.parent, delete=False) as handle:
temporary = Path(handle.name)
json.dump(report, handle, indent=2)
handle.write("\n")
try:
os.link(temporary, output)
finally:
temporary.unlink(missing_ok=True)
print(json.dumps({"output": str(output), "parity_passed": parity, "max_absolute_difference": report["max_absolute_difference"], "warm_serial": report["warm_serial"]["p95_ms"], "warm_batch": report["warm_batch"]["p95_ms"]}), flush=True)
if not parity:
raise SystemExit("Serial/batch parity failed; do not deploy")
if __name__ == "__main__":
main()