Record unseen music regressions and actual local reranker fidelity and cost
This commit is contained in:
karl committed
1 parent
cbf07b0575
commit
ec73d453e3
11 files changed
+1665
-1
No files matched your search
@@ -0,0 +1,208 @@
|
||||
# Local reranking experiment
|
||||
|
||||
Status: standalone CPU compatibility diagnostic executed; production integration
|
||||
and abstention remain unimplemented. Google EmbeddingGemma 2
|
||||
remains the mandatory document/code/image/audio/video retrieval model. Ordinary
|
||||
search remains local; this proposal adds no hosted inference or automatic downloads.
|
||||
Research checked 2026-10-13 using publisher metadata and the pinned native source.
|
||||
|
||||
## Smallest candidate worth testing
|
||||
|
||||
Try `cross-encoder/mmarco-mMiniLMv2-L12-H384-v1` as a **separate text reranker**.
|
||||
Its publisher identifies an Apache-2.0 multilingual query/passage classifier,
|
||||
trained on translated mMARCO. The dataset explicitly includes English and German.
|
||||
It does not replace Google embeddings or create another searchable vector index.
|
||||
[Publisher model card](https://huggingface.co/cross-encoder/mmarco-mMiniLMv2-L12-H384-v1),
|
||||
[dataset publisher](https://huggingface.co/datasets/unicamp-dl/mmarco).
|
||||
|
||||
The observed repository revision is
|
||||
`1427fd652930e4ba29e8149678df786c240d8825`. Its small configuration files report
|
||||
`XLMRobertaForSequenceClassification`, 12 layers, hidden size 384, vocabulary
|
||||
250002, one output label and tokenizer maximum length 512. Metadata was read;
|
||||
no model weights were downloaded.
|
||||
[Configuration](https://huggingface.co/cross-encoder/mmarco-mMiniLMv2-L12-H384-v1/blob/1427fd652930e4ba29e8149678df786c240d8825/config.json),
|
||||
[tokenizer configuration](https://huggingface.co/cross-encoder/mmarco-mMiniLMv2-L12-H384-v1/blob/1427fd652930e4ba29e8149678df786c240d8825/tokenizer_config.json).
|
||||
|
||||
Publisher artifacts provide an honest size comparison:
|
||||
|
||||
| Candidate | Publisher evidence | Disposition |
|
||||
| --- | --- | --- |
|
||||
| mMARCO multilingual MiniLM cross-encoder | FP32 safetensors 471 MB; SHA-256 `5daeca2481a76b5976a2bdc32f0a78532b6716da4f8cd3ff59460ef8d2f359b4` | First compatibility/cost experiment |
|
||||
| Same model's quantized ONNX AVX2 export | 119 MB; SHA-256 `6c2513767fb63d008a4377bef7a7a3555433d9436342bb53e35a3a72ffc52d4b` | Shows compact export is possible; adds another runtime if adopted |
|
||||
| BAAI bge-reranker-v2-m3 | Apache-2.0, FP32 safetensors 2.27 GB | Larger fallback research candidate; not the smallest first experiment |
|
||||
| Jina reranker v2 multilingual | Publisher restricts repository to CC-BY-NC-4.0 | Do not select for a broadly distributable default |
|
||||
|
||||
Artifact sources: [MiniLM safetensors](https://huggingface.co/cross-encoder/mmarco-mMiniLMv2-L12-H384-v1/blob/1427fd652930e4ba29e8149678df786c240d8825/model.safetensors),
|
||||
[MiniLM ONNX export](https://huggingface.co/cross-encoder/mmarco-mMiniLMv2-L12-H384-v1/blob/1427fd652930e4ba29e8149678df786c240d8825/onnx/model_quint8_avx2.onnx),
|
||||
[BAAI artifact](https://huggingface.co/BAAI/bge-reranker-v2-m3/blob/324cc40576b08b305b9c65a867c26c173a477ae2/model.safetensors),
|
||||
[Jina publisher licence](https://huggingface.co/jinaai/jina-reranker-v2-base-multilingual).
|
||||
The ONNX export names target different instruction families; no universal CPU
|
||||
compatibility or measured latency follows from the file names. Converted GGUF
|
||||
size, quantization fidelity and peak memory are still unknown.
|
||||
|
||||
## Reuse the native runtime first
|
||||
|
||||
The existing llama source revision
|
||||
`4d756bc72bf00a4aacf410ae15a2d315f3db400d` registers the candidate's actual
|
||||
architecture in `conversion/bert.py`. The embedding example includes query/document
|
||||
pair tokenization and `LLAMA_POOLING_TYPE_RANK`, with classification scores returned
|
||||
as rank outputs. Upstream documents a reranking endpoint and native CPU operation.
|
||||
This is source-level feasibility, **not a successful candidate conversion or execution**.
|
||||
[Native conversion source](https://github.com/ggml-org/llama.cpp/blob/4d756bc72bf00a4aacf410ae15a2d315f3db400d/conversion/bert.py),
|
||||
[native ranking example](https://github.com/ggml-org/llama.cpp/blob/4d756bc72bf00a4aacf410ae15a2d315f3db400d/examples/embedding/embedding.cpp),
|
||||
[upstream server contract](https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md).
|
||||
|
||||
Searchi's current worker intentionally validates 768-dimensional mean-pooled
|
||||
Google vectors. It cannot load this one-output classifier through its existing
|
||||
operations. Keep that invariant. First use the pinned upstream ranking example
|
||||
as a diagnostic rather than expanding the production worker protocol prematurely.
|
||||
The no-logits patch avoids unused vocabulary outputs for noncausal embedding
|
||||
contexts; classification-head rank outputs still need an actual compatibility test.
|
||||
Removing the patch does not turn EmbeddingGemma into an answer verifier.
|
||||
|
||||
Convert publisher-pinned weights with the pinned converter, without remote custom
|
||||
code. Record all tokenizer files, licence, sizes, SHA-256 values and conversion
|
||||
commands. Begin with F16 comparison, then Q8 only if reference parity is acceptable.
|
||||
Do not accept an unverified third-party GGUF merely because it loads. CPU execution
|
||||
and batch-versus-serial agreement must precede packaging or GPU claims.
|
||||
|
||||
## Bounded diagnostic, before product integration
|
||||
|
||||
1. Preserve Google candidate retrieval and all strict filters. Take at most five
|
||||
distinct source-verified text passages, one per selected file, with citations.
|
||||
Preserve exact identifiers and filename priority separately.
|
||||
2. Bound each query/passage pair to 256 tokenizer tokens, including special tokens;
|
||||
reserve at most 64 for the query and use actual token-aware passage truncation.
|
||||
Record truncated candidates. Do not treat truncated evidence as proof of absence.
|
||||
3. Compare serial one-thread CPU scoring with a bounded five-pair batch. Reuse one
|
||||
warmed session. Record pair construction, tokenization, inference and ordering
|
||||
separately; retain first-call, warm p95 and memory measurements.
|
||||
4. Predeclare a cost target: added warm p95 at most 200 ms and total warm search p95
|
||||
at most 500 ms on the declared reference machine. These are acceptance targets,
|
||||
not measurements. If unmet, keep the experiment out of default search.
|
||||
5. During diagnostics never suppress results. A cross-encoder's logit, or its
|
||||
sigmoid, is not a calibrated probability that the requested file exists.
|
||||
Failure, timeout, missing model or invalid output must preserve ordinary search.
|
||||
|
||||
A text classifier can evaluate documents, code, genuine tags and verified speech
|
||||
transcripts. It cannot determine whether a photograph contains an object or an
|
||||
instrumental recording contains a violin from placeholder media snippets. Visual
|
||||
and instrumental abstention require separate real multimodal evidence; this proposal
|
||||
does not close those gaps.
|
||||
|
||||
## Independent calibration and release decision
|
||||
|
||||
The old 0.70 cosine hypothesis failed held-out positive recall, and a correct
|
||||
German result scored below an absent request. Do not retune it or use the inspected
|
||||
mixed/relevance-v2 results as unseen validation. See [recorded failure](RELEVANCE-CALIBRATION.md).
|
||||
|
||||
Freeze a new small source corpus before scoring, with separate topic/repository
|
||||
families for calibration and held-out. Include English/German requests for present
|
||||
artifacts, absent concrete artifacts with closely related neighbours, exploratory
|
||||
queries, code implementations and negated/contradictory passages. Keep translated
|
||||
intent pairs together. Freeze labels and source bytes, obtain independent review,
|
||||
and record all Google raw candidates before reranking.
|
||||
|
||||
On calibration only, assess whether classifier scores plus retained retrieval
|
||||
signals improve false-match handling without losing weak-but-correct results.
|
||||
Predeclare any proposed suppression rule and its parameters, then evaluate held-out
|
||||
once. Report positive recall, absent-query return rate, hard negatives, truncation
|
||||
and latency together. A failed held-out policy is rejected, not adjusted on that
|
||||
same held-out set. Until evidence supports a default rule, keep related suggestions
|
||||
available and avoid a definitive no-match claim based solely on score.
|
||||
|
||||
## Actual CPU compatibility result
|
||||
|
||||
The native conversion route required absent Torch/Transformers dependencies.
|
||||
Instead of installing them, the existing CPU-only diagnostic environment provided
|
||||
ONNX Runtime 1.31.0, tokenizers 0.23.3 and NumPy 2.5.4 under Python 3.14.8. The
|
||||
publisher's pinned 118620016-byte ONNX export, 17082660-byte tokenizer and bounded
|
||||
configuration/card/licence files were downloaded and SHA-256 verified. No FP32
|
||||
weights, native conversion or production dependencies were added.
|
||||
|
||||
[Standalone tool](../tools/rerank-eval/README.md) froze its contract before inference:
|
||||
plan SHA-256 `02569e771ab74ff593842ff26d7eb42d1b0509bef09a1867cf0f1be6b1dc5a5a`,
|
||||
manifest SHA-256 `92ec98e8ccb5f8e8e08fbf46c7a4eb1407479b4dd0fafaabf6dc7cfc7aae2da7`.
|
||||
The five synthetic bilingual/text-code pairs are compatibility inputs, not an
|
||||
independent relevance corpus. Inference used only `CPUExecutionProvider`, one
|
||||
intra-op and one inter-op thread, sequential execution. The root agent's installer
|
||||
compression and Google package-startup checks finished before this inference slot;
|
||||
other unrelated VM load was not instrumented.
|
||||
|
||||
| Observation | Actual result |
|
||||
| --- | --- |
|
||||
| Asset verification / tokenizer setup | 93.30 / 506.61 ms |
|
||||
| ONNX session startup | 314.88 ms |
|
||||
| Pair token counts | 29, 37, 24, 27, 42 |
|
||||
| First five serial pairs / first batch | 51.99 / 76.09 ms |
|
||||
| Warm five-pair serial median / p95 | 52.42 / 70.93 ms |
|
||||
| Warm batch p95 | 67.25 ms |
|
||||
| Maximum serial/batch raw-logit difference | **0.880886** |
|
||||
| Frozen parity tolerance | absolute 0.001 plus relative 0.001 |
|
||||
| Disposition | **Parity failed; no deployment** |
|
||||
|
||||
Both score arrays are preserved; their ordering happens to agree on these five
|
||||
pairs. That does not excuse the failed score-invariance contract, especially for
|
||||
future rejection policies. Quantized batch effects are a possible explanation,
|
||||
not a verified cause. Do not relax the tolerance from these results. No confidence
|
||||
threshold was fitted or applied.
|
||||
|
||||
Ignored complete report `.searchi/rerank-eval/compatibility-v1.json`, SHA-256
|
||||
`a72cb628e2b635fe3549219d6be3065b9d3153225fd602e76502822a9db24e7c`.
|
||||
The process exited 1 after preserving this failed contract. No inference remains
|
||||
running. Timings cover short synthetic pairs and omit source retrieval, verification,
|
||||
admission, publication and UI costs; they do not prove the 256-token budget target
|
||||
or whole-pipeline performance. Peak private/resident memory was not measured.
|
||||
|
||||
Next, investigate score stability with a separately frozen serial-only diagnostic
|
||||
and an unquantized reference comparison before any calibration. Do not silently
|
||||
switch the existing plan's batch policy or overwrite this failure. Native runtime
|
||||
compatibility and Windows/Linux/macOS portability remain unverified for this model.
|
||||
|
||||
|
||||
## Unquantized reference follow-up
|
||||
|
||||
A separately frozen experiment verified the publisher F32 ONNX export at the same
|
||||
revision: 470883696 bytes, SHA-256
|
||||
`3e9a03ed1e966f7c5288dd4230e3d6a9bf5e3a170a06f1f4241c5bca12c6487c`.
|
||||
Its plan SHA-256 is
|
||||
`174867d4212b222e5302b1b16832b371c6bd38ec991f4145c952ed35446341f1`;
|
||||
manifest SHA-256 is
|
||||
`3c252dfc10e03ebba59a7ff0e33e6e2183a26c71862c52f0700f27ba659142be`.
|
||||
It retained the original five pairs and added one deliberately long pair, exercising
|
||||
64 query tokens and the complete 256-token pair budget. The original failed
|
||||
quantized report, tolerance and input plan were preserved.
|
||||
|
||||
F32 serial/batch parity **passed with zero observed logit difference** on these six
|
||||
inputs. The quantized single-pair outputs differed from F32 by up to 0.734820;
|
||||
ordering of the original five scores agreed. Quantized serial/batch differences
|
||||
remained observable at 0.378412 with the extra long input. This narrows the observed
|
||||
problem to the quantized export/execution path on this diagnostic; it does not
|
||||
identify the precise graph operator or prove general quantization fidelity.
|
||||
|
||||
| Six-pair F32 diagnostic | Actual observation |
|
||||
| --- | --- |
|
||||
| Asset verification / tokenizer setup | 388.83 / 497.59 ms |
|
||||
| Session startup | 1147.90 ms |
|
||||
| First serial / padded batch | 205.91 / 723.01 ms |
|
||||
| Warm serial median / p95 | 202.45 / 235.67 ms |
|
||||
| Warm padded batch p95 | 1017.59 ms |
|
||||
|
||||
The mixed-length batch padded all six pairs to 256 tokens, whereas serial execution
|
||||
retained each pair's actual length. These are different work volumes, not equivalent
|
||||
hardware scaling. Neither measured result establishes five full-budget passages
|
||||
under the proposed 200-ms target. The existing user app's Google query-worker
|
||||
startup may have overlapped the initial reference execution; unrelated VM contention
|
||||
was not instrumented. Treat these as diagnostic timings, not clean-room performance.
|
||||
|
||||
Ignored reference report `.searchi/rerank-eval/reference-v1.json`, SHA-256
|
||||
`de1d44245d6eb5fbf2a427b08b756b7c271b4daee57436155fa79ce9f684f100`.
|
||||
The process exited zero and owned inference ended. All executions explicitly used
|
||||
only CPU provider and one thread. No suppression, independent absence calibration,
|
||||
production integration, GPU or other-platform execution occurred.
|
||||
|
||||
The next useful experiment is new independent relevance calibration with deterministic
|
||||
single-pair scoring, while keeping the original mixed search results available.
|
||||
Default deployment still requires measured full-budget cost, memory, reference
|
||||
fidelity and positive-recall preservation. The batch failure is a reason to avoid
|
||||
using uncalibrated quantized batch logits for automatic no-answer decisions.
|
||||
@@ -84,4 +84,57 @@ lanes compare media-only, original tags, missing tags and misleading tags withou
|
||||
tuned weights or cutoff. Report positive recall and no-answer failure separately.
|
||||
This direct cosine diagnostic is not production ANN/hybrid validation.
|
||||
|
||||
The earlier prepared plan 8598312836f3c027b578f9f93ca1cd2d38d708bc0fd713fe174f10776934cbb2 was never inferred. Pre-inference review found that its verification decoded full float PCM while the production decoder uses millisecond trimming and s16 normalization. The corrected plan above matches the actual production decoder and compares only the whitelisted tag evidence; labels, source bytes, query texts and ranking hypothesis are unchanged. The earlier plan is preserved under its original ignored directory.
|
||||
The earlier prepared plan
|
||||
`8598312836f3c027b578f9f93ca1cd2d38d708bc0fd713fe174f10776934cbb2`
|
||||
was never inferred. Pre-inference review found that its verification decoded full
|
||||
float PCM while the production decoder uses millisecond trimming and s16
|
||||
normalization. The corrected plan above matches the actual production decoder
|
||||
and compares only the whitelisted tag evidence; labels, source bytes, query texts
|
||||
and ranking hypothesis are unchanged. The earlier plan is preserved under its
|
||||
original ignored directory.
|
||||
|
||||
## Actual frozen CPU results
|
||||
|
||||
The corrected policy completed from source `cbf07b0` without changing labels,
|
||||
competitors, weights or a cutoff. Independent read-only recomputation checked all
|
||||
18 queries, all four lanes, ten distinct competitors per lane and sorted scores.
|
||||
Missing-metadata rankings exactly equal the media-only rankings for every query.
|
||||
|
||||
| Direct file retrieval | Media only | Original tags | Missing tags | Misleading tags |
|
||||
| --- | ---: | ---: | ---: | ---: |
|
||||
| Correct first file, fourteen positives | 8/14 | 11/14 | 8/14 | 5/14 |
|
||||
| Recall in first five | 14/14 | 14/14 | 14/14 | 14/14 |
|
||||
| MRR | 0.726 | 0.869 | 0.726 | 0.570 |
|
||||
| No-answer queries returning candidates, failure | 4/4 | 4/4 | 4/4 | 4/4 |
|
||||
|
||||
Original fields improve both cello and both soprano-sax queries. However, the
|
||||
previously correct German acoustic-guitar query regresses from first to second:
|
||||
an unrelated tenor-sax title scores 0.65889 against the correct guitar audio's
|
||||
0.65581. Original tags leave both bucket-muted-trumpet queries incorrect. Misleading
|
||||
fields additionally lose both previously correct tenor-sax first hits and preserve
|
||||
the guitar regression. These losses are not hidden by aggregate gains or top-five
|
||||
recall. Four absent queries still return all ten candidates; no abstention
|
||||
improvement or rejection threshold follows.
|
||||
|
||||
Ten media encodes (eight audio sources and two photos) plus sixteen real tag-field
|
||||
encodes took **88.09 seconds** after model loading. That timer includes actual
|
||||
probing/decoding/source checks as well as inference; it excludes model startup and
|
||||
query inference and cannot isolate metadata's additional cost. The Windows 11
|
||||
10.0.26300 shared VM exposed eight logical CPUs and 16708012 KiB visible RAM on
|
||||
an AMD Ryzen 7 7800X3D host. Actual Google inference used one CPU thread, with the
|
||||
normal runtime forced to CPU. The installed app's warmed CPU query worker remained
|
||||
idle; no root model requests ran concurrently in the reserved inference slot.
|
||||
The native runtime continues to identify audio support as experimental.
|
||||
|
||||
The raw ignored report `.searchi/qa/music-validation-v1-canonical/results.json`
|
||||
contains 267667 bytes, SHA-256
|
||||
`69bcef5ffa58c0f23e5f34641256255bf15f54c836f75fb1aa878b4bd3df1309`.
|
||||
[Recomputed summary](evaluation-summary.json) preserves every query's first file,
|
||||
source kind/field, score and first relevant rank, plus worker/fixture hashes.
|
||||
|
||||
**Do not deploy the unconditional maximum policy as a universally better music
|
||||
ranking.** It is promising with informative correct tags, but this independently
|
||||
frozen sample exposes a real regression and sensitivity to misleading metadata.
|
||||
No policy was retuned after seeing these scores. Production ANN/hybrid evidence,
|
||||
more representative music, human listening, missing-content calibration and
|
||||
source-specific confidence remain open. The experimental media rating is unchanged.
|
||||
@@ -0,0 +1,844 @@
|
||||
{
|
||||
"source_commit": "cbf07b0",
|
||||
"fixture_sha256": "f9d7785ff937020b802e6c56252e64bbc3cd22e9a09709ddce9b82f9adc1b577",
|
||||
"prepared_sha256": "3e874108914e8f0334da8f12b704261e83755bd25dce6f93e22e8ce22b8dee78",
|
||||
"raw_report_sha256": "69bcef5ffa58c0f23e5f34641256255bf15f54c836f75fb1aa878b4bd3df1309",
|
||||
"raw_report_bytes": 267667,
|
||||
"worker_sha256": "c01cb3ec13cfca722ee768e8350e1399a8b9a4d14a24a35adc3473a5cfc82a73",
|
||||
"worker": {
|
||||
"code_query_supported": true,
|
||||
"context_tokens": 2048,
|
||||
"device": "cpu",
|
||||
"dimensions": 768,
|
||||
"media_supported": true,
|
||||
"model": "google/embeddinggemma-2",
|
||||
"ready": true,
|
||||
"role": "indexing",
|
||||
"runtime": "b11461-no-logits-v1",
|
||||
"threads": 1
|
||||
},
|
||||
"encoding_ms": 88089.5138,
|
||||
"media_inference_calls": 10,
|
||||
"metadata_inference_calls": 16,
|
||||
"metrics": {
|
||||
"audio_only": {
|
||||
"positive_queries": 14,
|
||||
"top1": 8,
|
||||
"recall_at_5": 14,
|
||||
"mrr": 0.7261904761904762,
|
||||
"no_answer_queries": 4,
|
||||
"no_answer_returned_any": 4
|
||||
},
|
||||
"original": {
|
||||
"positive_queries": 14,
|
||||
"top1": 11,
|
||||
"recall_at_5": 14,
|
||||
"mrr": 0.869047619047619,
|
||||
"no_answer_queries": 4,
|
||||
"no_answer_returned_any": 4
|
||||
},
|
||||
"missing_metadata": {
|
||||
"positive_queries": 14,
|
||||
"top1": 8,
|
||||
"recall_at_5": 14,
|
||||
"mrr": 0.7261904761904762,
|
||||
"no_answer_queries": 4,
|
||||
"no_answer_returned_any": 4
|
||||
},
|
||||
"misleading_metadata": {
|
||||
"positive_queries": 14,
|
||||
"top1": 5,
|
||||
"recall_at_5": 14,
|
||||
"mrr": 0.5702380952380952,
|
||||
"no_answer_queries": 4,
|
||||
"no_answer_returned_any": 4
|
||||
}
|
||||
},
|
||||
"changes_vs_audio": {
|
||||
"original": {
|
||||
"top1_gains_vs_audio": [
|
||||
"unseen02-en",
|
||||
"unseen02-de",
|
||||
"unseen06-en",
|
||||
"unseen06-de"
|
||||
],
|
||||
"top1_losses_vs_audio": [
|
||||
"unseen01-de"
|
||||
]
|
||||
},
|
||||
"missing_metadata": {
|
||||
"top1_gains_vs_audio": [],
|
||||
"top1_losses_vs_audio": []
|
||||
},
|
||||
"misleading_metadata": {
|
||||
"top1_gains_vs_audio": [],
|
||||
"top1_losses_vs_audio": [
|
||||
"unseen01-de",
|
||||
"unseen05-en",
|
||||
"unseen05-de"
|
||||
]
|
||||
}
|
||||
},
|
||||
"rows": [
|
||||
{
|
||||
"id": "unseen01-en",
|
||||
"expected": [
|
||||
"unseen01"
|
||||
],
|
||||
"language": "en",
|
||||
"category": "instrument",
|
||||
"lanes": {
|
||||
"audio_only": {
|
||||
"first_relevant_rank": 1,
|
||||
"returned_count": 10,
|
||||
"first_file": "unseen01",
|
||||
"first_source_kind": "audio",
|
||||
"first_source_field": null,
|
||||
"first_score": 0.6707203058852982
|
||||
},
|
||||
"original": {
|
||||
"first_relevant_rank": 1,
|
||||
"returned_count": 10,
|
||||
"first_file": "unseen01",
|
||||
"first_source_kind": "audio",
|
||||
"first_source_field": null,
|
||||
"first_score": 0.6707203058852982
|
||||
},
|
||||
"missing_metadata": {
|
||||
"first_relevant_rank": 1,
|
||||
"returned_count": 10,
|
||||
"first_file": "unseen01",
|
||||
"first_source_kind": "audio",
|
||||
"first_source_field": null,
|
||||
"first_score": 0.6707203058852982
|
||||
},
|
||||
"misleading_metadata": {
|
||||
"first_relevant_rank": 1,
|
||||
"returned_count": 10,
|
||||
"first_file": "unseen01",
|
||||
"first_source_kind": "audio",
|
||||
"first_source_field": null,
|
||||
"first_score": 0.6707203058852982
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "unseen01-de",
|
||||
"expected": [
|
||||
"unseen01"
|
||||
],
|
||||
"language": "de",
|
||||
"category": "instrument",
|
||||
"lanes": {
|
||||
"audio_only": {
|
||||
"first_relevant_rank": 1,
|
||||
"returned_count": 10,
|
||||
"first_file": "unseen01",
|
||||
"first_source_kind": "audio",
|
||||
"first_source_field": null,
|
||||
"first_score": 0.6558122105336331
|
||||
},
|
||||
"original": {
|
||||
"first_relevant_rank": 2,
|
||||
"returned_count": 10,
|
||||
"first_file": "unseen05",
|
||||
"first_source_kind": "metadata",
|
||||
"first_source_field": "title",
|
||||
"first_score": 0.6588859671020654
|
||||
},
|
||||
"missing_metadata": {
|
||||
"first_relevant_rank": 1,
|
||||
"returned_count": 10,
|
||||
"first_file": "unseen01",
|
||||
"first_source_kind": "audio",
|
||||
"first_source_field": null,
|
||||
"first_score": 0.6558122105336331
|
||||
},
|
||||
"misleading_metadata": {
|
||||
"first_relevant_rank": 2,
|
||||
"returned_count": 10,
|
||||
"first_file": "unseen04",
|
||||
"first_source_kind": "metadata",
|
||||
"first_source_field": "title",
|
||||
"first_score": 0.6588859671020654
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "unseen02-en",
|
||||
"expected": [
|
||||
"unseen02"
|
||||
],
|
||||
"language": "en",
|
||||
"category": "instrument",
|
||||
"lanes": {
|
||||
"audio_only": {
|
||||
"first_relevant_rank": 2,
|
||||
"returned_count": 10,
|
||||
"first_file": "unseen05",
|
||||
"first_source_kind": "audio",
|
||||
"first_source_field": null,
|
||||
"first_score": 0.6690030556732439
|
||||
},
|
||||
"original": {
|
||||
"first_relevant_rank": 1,
|
||||
"returned_count": 10,
|
||||
"first_file": "unseen02",
|
||||
"first_source_kind": "metadata",
|
||||
"first_source_field": "title",
|
||||
"first_score": 0.7723038021644638
|
||||
},
|
||||
"missing_metadata": {
|
||||
"first_relevant_rank": 2,
|
||||
"returned_count": 10,
|
||||
"first_file": "unseen05",
|
||||
"first_source_kind": "audio",
|
||||
"first_source_field": null,
|
||||
"first_score": 0.6690030556732439
|
||||
},
|
||||
"misleading_metadata": {
|
||||
"first_relevant_rank": 4,
|
||||
"returned_count": 10,
|
||||
"first_file": "unseen01",
|
||||
"first_source_kind": "metadata",
|
||||
"first_source_field": "title",
|
||||
"first_score": 0.7723038021644638
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "unseen02-de",
|
||||
"expected": [
|
||||
"unseen02"
|
||||
],
|
||||
"language": "de",
|
||||
"category": "instrument",
|
||||
"lanes": {
|
||||
"audio_only": {
|
||||
"first_relevant_rank": 2,
|
||||
"returned_count": 10,
|
||||
"first_file": "unseen05",
|
||||
"first_source_kind": "audio",
|
||||
"first_source_field": null,
|
||||
"first_score": 0.667712688085295
|
||||
},
|
||||
"original": {
|
||||
"first_relevant_rank": 1,
|
||||
"returned_count": 10,
|
||||
"first_file": "unseen02",
|
||||
"first_source_kind": "metadata",
|
||||
"first_source_field": "title",
|
||||
"first_score": 0.7157722798891778
|
||||
},
|
||||
"missing_metadata": {
|
||||
"first_relevant_rank": 2,
|
||||
"returned_count": 10,
|
||||
"first_file": "unseen05",
|
||||
"first_source_kind": "audio",
|
||||
"first_source_field": null,
|
||||
"first_score": 0.667712688085295
|
||||
},
|
||||
"misleading_metadata": {
|
||||
"first_relevant_rank": 4,
|
||||
"returned_count": 10,
|
||||
"first_file": "unseen01",
|
||||
"first_source_kind": "metadata",
|
||||
"first_source_field": "title",
|
||||
"first_score": 0.7157722798891778
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "unseen03-en",
|
||||
"expected": [
|
||||
"unseen03"
|
||||
],
|
||||
"language": "en",
|
||||
"category": "instrument",
|
||||
"lanes": {
|
||||
"audio_only": {
|
||||
"first_relevant_rank": 1,
|
||||
"returned_count": 10,
|
||||
"first_file": "unseen03",
|
||||
"first_source_kind": "audio",
|
||||
"first_source_field": null,
|
||||
"first_score": 0.6901545281005906
|
||||
},
|
||||
"original": {
|
||||
"first_relevant_rank": 1,
|
||||
"returned_count": 10,
|
||||
"first_file": "unseen03",
|
||||
"first_source_kind": "audio",
|
||||
"first_source_field": null,
|
||||
"first_score": 0.6901545281005906
|
||||
},
|
||||
"missing_metadata": {
|
||||
"first_relevant_rank": 1,
|
||||
"returned_count": 10,
|
||||
"first_file": "unseen03",
|
||||
"first_source_kind": "audio",
|
||||
"first_source_field": null,
|
||||
"first_score": 0.6901545281005906
|
||||
},
|
||||
"misleading_metadata": {
|
||||
"first_relevant_rank": 1,
|
||||
"returned_count": 10,
|
||||
"first_file": "unseen03",
|
||||
"first_source_kind": "audio",
|
||||
"first_source_field": null,
|
||||
"first_score": 0.6901545281005906
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "unseen03-de",
|
||||
"expected": [
|
||||
"unseen03"
|
||||
],
|
||||
"language": "de",
|
||||
"category": "instrument",
|
||||
"lanes": {
|
||||
"audio_only": {
|
||||
"first_relevant_rank": 1,
|
||||
"returned_count": 10,
|
||||
"first_file": "unseen03",
|
||||
"first_source_kind": "audio",
|
||||
"first_source_field": null,
|
||||
"first_score": 0.6759388311775514
|
||||
},
|
||||
"original": {
|
||||
"first_relevant_rank": 1,
|
||||
"returned_count": 10,
|
||||
"first_file": "unseen03",
|
||||
"first_source_kind": "audio",
|
||||
"first_source_field": null,
|
||||
"first_score": 0.6759388311775514
|
||||
},
|
||||
"missing_metadata": {
|
||||
"first_relevant_rank": 1,
|
||||
"returned_count": 10,
|
||||
"first_file": "unseen03",
|
||||
"first_source_kind": "audio",
|
||||
"first_source_field": null,
|
||||
"first_score": 0.6759388311775514
|
||||
},
|
||||
"misleading_metadata": {
|
||||
"first_relevant_rank": 1,
|
||||
"returned_count": 10,
|
||||
"first_file": "unseen03",
|
||||
"first_source_kind": "audio",
|
||||
"first_source_field": null,
|
||||
"first_score": 0.6759388311775514
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "unseen04-en",
|
||||
"expected": [
|
||||
"unseen04"
|
||||
],
|
||||
"language": "en",
|
||||
"category": "instrument",
|
||||
"lanes": {
|
||||
"audio_only": {
|
||||
"first_relevant_rank": 3,
|
||||
"returned_count": 10,
|
||||
"first_file": "unseen05",
|
||||
"first_source_kind": "audio",
|
||||
"first_source_field": null,
|
||||
"first_score": 0.7212670979690754
|
||||
},
|
||||
"original": {
|
||||
"first_relevant_rank": 3,
|
||||
"returned_count": 10,
|
||||
"first_file": "unseen05",
|
||||
"first_source_kind": "audio",
|
||||
"first_source_field": null,
|
||||
"first_score": 0.7212670979690754
|
||||
},
|
||||
"missing_metadata": {
|
||||
"first_relevant_rank": 3,
|
||||
"returned_count": 10,
|
||||
"first_file": "unseen05",
|
||||
"first_source_kind": "audio",
|
||||
"first_source_field": null,
|
||||
"first_score": 0.7212670979690754
|
||||
},
|
||||
"misleading_metadata": {
|
||||
"first_relevant_rank": 4,
|
||||
"returned_count": 10,
|
||||
"first_file": "unseen05",
|
||||
"first_source_kind": "audio",
|
||||
"first_source_field": null,
|
||||
"first_score": 0.7212670979690754
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "unseen04-de",
|
||||
"expected": [
|
||||
"unseen04"
|
||||
],
|
||||
"language": "de",
|
||||
"category": "instrument",
|
||||
"lanes": {
|
||||
"audio_only": {
|
||||
"first_relevant_rank": 3,
|
||||
"returned_count": 10,
|
||||
"first_file": "unseen05",
|
||||
"first_source_kind": "audio",
|
||||
"first_source_field": null,
|
||||
"first_score": 0.7094035244502205
|
||||
},
|
||||
"original": {
|
||||
"first_relevant_rank": 3,
|
||||
"returned_count": 10,
|
||||
"first_file": "unseen05",
|
||||
"first_source_kind": "audio",
|
||||
"first_source_field": null,
|
||||
"first_score": 0.7094035244502205
|
||||
},
|
||||
"missing_metadata": {
|
||||
"first_relevant_rank": 3,
|
||||
"returned_count": 10,
|
||||
"first_file": "unseen05",
|
||||
"first_source_kind": "audio",
|
||||
"first_source_field": null,
|
||||
"first_score": 0.7094035244502205
|
||||
},
|
||||
"misleading_metadata": {
|
||||
"first_relevant_rank": 3,
|
||||
"returned_count": 10,
|
||||
"first_file": "unseen05",
|
||||
"first_source_kind": "audio",
|
||||
"first_source_field": null,
|
||||
"first_score": 0.7094035244502205
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "unseen05-en",
|
||||
"expected": [
|
||||
"unseen05"
|
||||
],
|
||||
"language": "en",
|
||||
"category": "instrument",
|
||||
"lanes": {
|
||||
"audio_only": {
|
||||
"first_relevant_rank": 1,
|
||||
"returned_count": 10,
|
||||
"first_file": "unseen05",
|
||||
"first_source_kind": "audio",
|
||||
"first_source_field": null,
|
||||
"first_score": 0.6986278621913614
|
||||
},
|
||||
"original": {
|
||||
"first_relevant_rank": 1,
|
||||
"returned_count": 10,
|
||||
"first_file": "unseen05",
|
||||
"first_source_kind": "metadata",
|
||||
"first_source_field": "title",
|
||||
"first_score": 0.7804412810940734
|
||||
},
|
||||
"missing_metadata": {
|
||||
"first_relevant_rank": 1,
|
||||
"returned_count": 10,
|
||||
"first_file": "unseen05",
|
||||
"first_source_kind": "audio",
|
||||
"first_source_field": null,
|
||||
"first_score": 0.6986278621913614
|
||||
},
|
||||
"misleading_metadata": {
|
||||
"first_relevant_rank": 2,
|
||||
"returned_count": 10,
|
||||
"first_file": "unseen04",
|
||||
"first_source_kind": "metadata",
|
||||
"first_source_field": "title",
|
||||
"first_score": 0.7804412810940734
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "unseen05-de",
|
||||
"expected": [
|
||||
"unseen05"
|
||||
],
|
||||
"language": "de",
|
||||
"category": "instrument",
|
||||
"lanes": {
|
||||
"audio_only": {
|
||||
"first_relevant_rank": 1,
|
||||
"returned_count": 10,
|
||||
"first_file": "unseen05",
|
||||
"first_source_kind": "audio",
|
||||
"first_source_field": null,
|
||||
"first_score": 0.7104005169326468
|
||||
},
|
||||
"original": {
|
||||
"first_relevant_rank": 1,
|
||||
"returned_count": 10,
|
||||
"first_file": "unseen05",
|
||||
"first_source_kind": "metadata",
|
||||
"first_source_field": "title",
|
||||
"first_score": 0.7544482777868109
|
||||
},
|
||||
"missing_metadata": {
|
||||
"first_relevant_rank": 1,
|
||||
"returned_count": 10,
|
||||
"first_file": "unseen05",
|
||||
"first_source_kind": "audio",
|
||||
"first_source_field": null,
|
||||
"first_score": 0.7104005169326468
|
||||
},
|
||||
"misleading_metadata": {
|
||||
"first_relevant_rank": 2,
|
||||
"returned_count": 10,
|
||||
"first_file": "unseen04",
|
||||
"first_source_kind": "metadata",
|
||||
"first_source_field": "title",
|
||||
"first_score": 0.7544482777868109
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "unseen06-en",
|
||||
"expected": [
|
||||
"unseen06"
|
||||
],
|
||||
"language": "en",
|
||||
"category": "instrument",
|
||||
"lanes": {
|
||||
"audio_only": {
|
||||
"first_relevant_rank": 4,
|
||||
"returned_count": 10,
|
||||
"first_file": "unseen05",
|
||||
"first_source_kind": "audio",
|
||||
"first_source_field": null,
|
||||
"first_score": 0.7048521660816882
|
||||
},
|
||||
"original": {
|
||||
"first_relevant_rank": 1,
|
||||
"returned_count": 10,
|
||||
"first_file": "unseen06",
|
||||
"first_source_kind": "metadata",
|
||||
"first_source_field": "title",
|
||||
"first_score": 0.7621619204850741
|
||||
},
|
||||
"missing_metadata": {
|
||||
"first_relevant_rank": 4,
|
||||
"returned_count": 10,
|
||||
"first_file": "unseen05",
|
||||
"first_source_kind": "audio",
|
||||
"first_source_field": null,
|
||||
"first_score": 0.7048521660816882
|
||||
},
|
||||
"misleading_metadata": {
|
||||
"first_relevant_rank": 5,
|
||||
"returned_count": 10,
|
||||
"first_file": "unseen05",
|
||||
"first_source_kind": "metadata",
|
||||
"first_source_field": "title",
|
||||
"first_score": 0.7621619204850741
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "unseen06-de",
|
||||
"expected": [
|
||||
"unseen06"
|
||||
],
|
||||
"language": "de",
|
||||
"category": "instrument",
|
||||
"lanes": {
|
||||
"audio_only": {
|
||||
"first_relevant_rank": 4,
|
||||
"returned_count": 10,
|
||||
"first_file": "unseen05",
|
||||
"first_source_kind": "audio",
|
||||
"first_source_field": null,
|
||||
"first_score": 0.7064829076815999
|
||||
},
|
||||
"original": {
|
||||
"first_relevant_rank": 1,
|
||||
"returned_count": 10,
|
||||
"first_file": "unseen06",
|
||||
"first_source_kind": "metadata",
|
||||
"first_source_field": "title",
|
||||
"first_score": 0.7392143518999531
|
||||
},
|
||||
"missing_metadata": {
|
||||
"first_relevant_rank": 4,
|
||||
"returned_count": 10,
|
||||
"first_file": "unseen05",
|
||||
"first_source_kind": "audio",
|
||||
"first_source_field": null,
|
||||
"first_score": 0.7064829076815999
|
||||
},
|
||||
"misleading_metadata": {
|
||||
"first_relevant_rank": 5,
|
||||
"returned_count": 10,
|
||||
"first_file": "unseen05",
|
||||
"first_source_kind": "metadata",
|
||||
"first_source_field": "title",
|
||||
"first_score": 0.7392143518999531
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "sax-broad-en",
|
||||
"expected": [
|
||||
"unseen05",
|
||||
"unseen06"
|
||||
],
|
||||
"language": "en",
|
||||
"category": "related-positive",
|
||||
"lanes": {
|
||||
"audio_only": {
|
||||
"first_relevant_rank": 1,
|
||||
"returned_count": 10,
|
||||
"first_file": "unseen05",
|
||||
"first_source_kind": "audio",
|
||||
"first_source_field": null,
|
||||
"first_score": 0.7506144568545938
|
||||
},
|
||||
"original": {
|
||||
"first_relevant_rank": 1,
|
||||
"returned_count": 10,
|
||||
"first_file": "unseen06",
|
||||
"first_source_kind": "metadata",
|
||||
"first_source_field": "title",
|
||||
"first_score": 0.7558704108318935
|
||||
},
|
||||
"missing_metadata": {
|
||||
"first_relevant_rank": 1,
|
||||
"returned_count": 10,
|
||||
"first_file": "unseen05",
|
||||
"first_source_kind": "audio",
|
||||
"first_source_field": null,
|
||||
"first_score": 0.7506144568545938
|
||||
},
|
||||
"misleading_metadata": {
|
||||
"first_relevant_rank": 1,
|
||||
"returned_count": 10,
|
||||
"first_file": "unseen05",
|
||||
"first_source_kind": "metadata",
|
||||
"first_source_field": "title",
|
||||
"first_score": 0.7558704108318935
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "sax-broad-de",
|
||||
"expected": [
|
||||
"unseen05",
|
||||
"unseen06"
|
||||
],
|
||||
"language": "de",
|
||||
"category": "related-positive",
|
||||
"lanes": {
|
||||
"audio_only": {
|
||||
"first_relevant_rank": 1,
|
||||
"returned_count": 10,
|
||||
"first_file": "unseen05",
|
||||
"first_source_kind": "audio",
|
||||
"first_source_field": null,
|
||||
"first_score": 0.7499595252586609
|
||||
},
|
||||
"original": {
|
||||
"first_relevant_rank": 1,
|
||||
"returned_count": 10,
|
||||
"first_file": "unseen06",
|
||||
"first_source_kind": "metadata",
|
||||
"first_source_field": "title",
|
||||
"first_score": 0.762964312138735
|
||||
},
|
||||
"missing_metadata": {
|
||||
"first_relevant_rank": 1,
|
||||
"returned_count": 10,
|
||||
"first_file": "unseen05",
|
||||
"first_source_kind": "audio",
|
||||
"first_source_field": null,
|
||||
"first_score": 0.7499595252586609
|
||||
},
|
||||
"misleading_metadata": {
|
||||
"first_relevant_rank": 1,
|
||||
"returned_count": 10,
|
||||
"first_file": "unseen05",
|
||||
"first_source_kind": "metadata",
|
||||
"first_source_field": "title",
|
||||
"first_score": 0.762964312138735
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "absent-bagpipe-en",
|
||||
"expected": [],
|
||||
"language": "en",
|
||||
"category": "absent-instrument",
|
||||
"lanes": {
|
||||
"audio_only": {
|
||||
"first_relevant_rank": null,
|
||||
"returned_count": 10,
|
||||
"first_file": "unseen05",
|
||||
"first_source_kind": "audio",
|
||||
"first_source_field": null,
|
||||
"first_score": 0.6899061391978294
|
||||
},
|
||||
"original": {
|
||||
"first_relevant_rank": null,
|
||||
"returned_count": 10,
|
||||
"first_file": "unseen05",
|
||||
"first_source_kind": "audio",
|
||||
"first_source_field": null,
|
||||
"first_score": 0.6899061391978294
|
||||
},
|
||||
"missing_metadata": {
|
||||
"first_relevant_rank": null,
|
||||
"returned_count": 10,
|
||||
"first_file": "unseen05",
|
||||
"first_source_kind": "audio",
|
||||
"first_source_field": null,
|
||||
"first_score": 0.6899061391978294
|
||||
},
|
||||
"misleading_metadata": {
|
||||
"first_relevant_rank": null,
|
||||
"returned_count": 10,
|
||||
"first_file": "unseen05",
|
||||
"first_source_kind": "audio",
|
||||
"first_source_field": null,
|
||||
"first_score": 0.6899061391978294
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "absent-bagpipe-de",
|
||||
"expected": [],
|
||||
"language": "de",
|
||||
"category": "absent-instrument",
|
||||
"lanes": {
|
||||
"audio_only": {
|
||||
"first_relevant_rank": null,
|
||||
"returned_count": 10,
|
||||
"first_file": "unseen05",
|
||||
"first_source_kind": "audio",
|
||||
"first_source_field": null,
|
||||
"first_score": 0.6517667406795858
|
||||
},
|
||||
"original": {
|
||||
"first_relevant_rank": null,
|
||||
"returned_count": 10,
|
||||
"first_file": "unseen05",
|
||||
"first_source_kind": "audio",
|
||||
"first_source_field": null,
|
||||
"first_score": 0.6517667406795858
|
||||
},
|
||||
"missing_metadata": {
|
||||
"first_relevant_rank": null,
|
||||
"returned_count": 10,
|
||||
"first_file": "unseen05",
|
||||
"first_source_kind": "audio",
|
||||
"first_source_field": null,
|
||||
"first_score": 0.6517667406795858
|
||||
},
|
||||
"misleading_metadata": {
|
||||
"first_relevant_rank": null,
|
||||
"returned_count": 10,
|
||||
"first_file": "unseen05",
|
||||
"first_source_kind": "audio",
|
||||
"first_source_field": null,
|
||||
"first_score": 0.6517667406795858
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "absent-artist-en",
|
||||
"expected": [],
|
||||
"language": "en",
|
||||
"category": "absent-identity",
|
||||
"lanes": {
|
||||
"audio_only": {
|
||||
"first_relevant_rank": null,
|
||||
"returned_count": 10,
|
||||
"first_file": "sound02",
|
||||
"first_source_kind": "audio",
|
||||
"first_source_field": null,
|
||||
"first_score": 0.5977197385044184
|
||||
},
|
||||
"original": {
|
||||
"first_relevant_rank": null,
|
||||
"returned_count": 10,
|
||||
"first_file": "unseen05",
|
||||
"first_source_kind": "metadata",
|
||||
"first_source_field": "title",
|
||||
"first_score": 0.5983290055413841
|
||||
},
|
||||
"missing_metadata": {
|
||||
"first_relevant_rank": null,
|
||||
"returned_count": 10,
|
||||
"first_file": "sound02",
|
||||
"first_source_kind": "audio",
|
||||
"first_source_field": null,
|
||||
"first_score": 0.5977197385044184
|
||||
},
|
||||
"misleading_metadata": {
|
||||
"first_relevant_rank": null,
|
||||
"returned_count": 10,
|
||||
"first_file": "unseen04",
|
||||
"first_source_kind": "metadata",
|
||||
"first_source_field": "title",
|
||||
"first_score": 0.5983290055413841
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "absent-content-en",
|
||||
"expected": [],
|
||||
"language": "en",
|
||||
"category": "absent-content",
|
||||
"lanes": {
|
||||
"audio_only": {
|
||||
"first_relevant_rank": null,
|
||||
"returned_count": 10,
|
||||
"first_file": "unseen01",
|
||||
"first_source_kind": "audio",
|
||||
"first_source_field": null,
|
||||
"first_score": 0.5870996954366186
|
||||
},
|
||||
"original": {
|
||||
"first_relevant_rank": null,
|
||||
"returned_count": 10,
|
||||
"first_file": "unseen01",
|
||||
"first_source_kind": "audio",
|
||||
"first_source_field": null,
|
||||
"first_score": 0.5870996954366186
|
||||
},
|
||||
"missing_metadata": {
|
||||
"first_relevant_rank": null,
|
||||
"returned_count": 10,
|
||||
"first_file": "unseen01",
|
||||
"first_source_kind": "audio",
|
||||
"first_source_field": null,
|
||||
"first_score": 0.5870996954366186
|
||||
},
|
||||
"misleading_metadata": {
|
||||
"first_relevant_rank": null,
|
||||
"returned_count": 10,
|
||||
"first_file": "unseen01",
|
||||
"first_source_kind": "audio",
|
||||
"first_source_field": null,
|
||||
"first_score": 0.5870996954366186
|
||||
}
|
||||
}
|
||||
}
|
||||
],
|
||||
"environment": {
|
||||
"os": "Windows11 Pro10.0.26300",
|
||||
"logical_cpus": 8,
|
||||
"visible_memory_kib": 16708012,
|
||||
"processor": "AMD Ryzen7 7800X3D",
|
||||
"shared_vm": true,
|
||||
"existing_app_query_worker": "Warmed CPU session remained idle; no root model requests during reserved slot"
|
||||
},
|
||||
"production_policy_changed": false,
|
||||
"deployment_conclusion": "Original metadata improves aggregate retrieval but introduces a previously-correct German guitar failure; misleading metadata reduces top1 and no-answer remains unsolved. Unconditional max policy not validated for deployment. No post-score policy tuning."
|
||||
}
|
||||
@@ -0,0 +1,44 @@
|
||||
# Local cross-encoder diagnostic
|
||||
|
||||
This standalone research tool never modifies production search or its Google
|
||||
EmbeddingGemma 2 model. Do not confuse the cross-encoder's name with the historical
|
||||
MiniLM embedding experiment. Publisher weights stay under ignored `.searchi` data.
|
||||
|
||||
Pinned publisher revision, asset bytes/SHA-256 and licence provenance are in
|
||||
`model-manifest.json`; the pre-inference contract is in `compatibility-plan.json`.
|
||||
The existing CPU Python environment can execute it without installing Torch or
|
||||
changing production dependencies:
|
||||
|
||||
```powershell
|
||||
python tools/rerank-eval/download.py
|
||||
$env:SEARCHI_FORCE_CPU = '1'
|
||||
& tools/semantic-proof/.venv/Scripts/python.exe tools/rerank-eval/compatibility.py --output .searchi/rerank-eval/compatibility-v1.json
|
||||
```
|
||||
|
||||
The runner verifies all assets, rejects changed frozen inputs, enables only CPU
|
||||
Execution Provider with one thread, bounds five pairs to 256 tokens including
|
||||
special tokens and at most 64 query tokens, checks finite one-logit-per-pair
|
||||
outputs, and records serial/batch parity and twenty warm timing samples. It creates
|
||||
new reports atomically and refuses overwriting prior evidence. A failed parity
|
||||
contract writes the observed report then exits nonzero. Never weaken its frozen
|
||||
tolerance after observing output.
|
||||
|
||||
The first execution failed serial/batch logit parity; see
|
||||
[recorded result](../../docs/LOCAL-RERANKING.md). It is not a product integration,
|
||||
relevance calibration or full search performance benchmark. Scores are not
|
||||
probabilities that the requested artifact exists. Any further experimental input
|
||||
or scoring policy needs a separate version and output path.
|
||||
|
||||
The separate unquantized follow-up preserves the first failed evidence:
|
||||
|
||||
```powershell
|
||||
python tools/rerank-eval/download.py --reference
|
||||
$env:SEARCHI_FORCE_CPU = '1'
|
||||
& tools/semantic-proof/.venv/Scripts/python.exe tools/rerank-eval/reference.py --output .searchi/rerank-eval/reference-v1.json
|
||||
```
|
||||
|
||||
`reference-plan.json` retains five short pairs and adds a 256-token budget case.
|
||||
F32 parity passed; quantized score differences persisted. Mixed-length padded
|
||||
batching was slower. Neither result establishes general absence calibration or
|
||||
sufficient default-search latency. Detailed costs and possible VM contention are
|
||||
recorded in the linked report documentation.
|
||||
@@ -0,0 +1,38 @@
|
||||
{
|
||||
"version": 1,
|
||||
"status": "pre-inference frozen compatibility/cost contract, not relevance calibration",
|
||||
"cpu_provider": "CPUExecutionProvider",
|
||||
"intra_op_threads": 1,
|
||||
"inter_op_threads": 1,
|
||||
"parallel_execution": false,
|
||||
"max_pairs": 5,
|
||||
"max_pair_tokens": 256,
|
||||
"max_query_tokens": 64,
|
||||
"warm_iterations": 20,
|
||||
"parity_absolute_tolerance": 0.001,
|
||||
"parity_relative_tolerance": 0.001,
|
||||
"added_warm_p95_target_ms": 200,
|
||||
"no_suppression": true,
|
||||
"pairs": [
|
||||
[
|
||||
"How do we cancel the supplier partnership?",
|
||||
"Either party may terminate the supplier contract by giving sixty days written notice."
|
||||
],
|
||||
[
|
||||
"Wie beenden wir die Zusammenarbeit mit dem Lieferanten?",
|
||||
"Jede Partei kann die Zusammenarbeit mit dem Lieferanten mit einer Frist von sechzig Tagen kündigen."
|
||||
],
|
||||
[
|
||||
"How do we cancel the supplier partnership?",
|
||||
"Tomatoes require sunshine and consistent watering."
|
||||
],
|
||||
[
|
||||
"Wie beenden wir die Zusammenarbeit mit dem Lieferanten?",
|
||||
"Der Garten benötigt Sonnenlicht und regelmäßige Bewässerung."
|
||||
],
|
||||
[
|
||||
"Where does the code implement retry backoff?",
|
||||
"fn retry_delay(attempt: u32) -> u64 { 100 * (1 << attempt.min(6)) }"
|
||||
]
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,131 @@
|
||||
"""Owned CPU-only cross-encoder compatibility/cost diagnostic; never alters search."""
|
||||
import argparse
|
||||
import hashlib
|
||||
import json
|
||||
import math
|
||||
import os
|
||||
from pathlib import Path
|
||||
import platform
|
||||
import statistics
|
||||
import tempfile
|
||||
import time
|
||||
|
||||
import numpy as np
|
||||
import onnxruntime as ort
|
||||
from tokenizers import Tokenizer
|
||||
from download import DESTINATION, digest
|
||||
|
||||
EXPECTED_PLAN = "02569e771ab74ff593842ff26d7eb42d1b0509bef09a1867cf0f1be6b1dc5a5a"
|
||||
EXPECTED_MANIFEST = "92ec98e8ccb5f8e8e08fbf46c7a4eb1407479b4dd0fafaabf6dc7cfc7aae2da7"
|
||||
|
||||
|
||||
def checked_json(path, expected):
|
||||
data = path.read_bytes()
|
||||
if hashlib.sha256(data).hexdigest() != expected:
|
||||
raise ValueError("Frozen diagnostic inputs changed")
|
||||
return json.loads(data)
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("--output", type=Path, required=True)
|
||||
args = parser.parse_args()
|
||||
owned_root = DESTINATION.parents[1].resolve()
|
||||
output = args.output.resolve()
|
||||
if not output.is_relative_to(owned_root) or output.exists():
|
||||
raise ValueError("Output must be new and under owned .searchi data")
|
||||
if os.environ.get("SEARCHI_FORCE_CPU") != "1":
|
||||
raise ValueError("Windows VM diagnostic requires explicit SEARCHI_FORCE_CPU=1")
|
||||
plan = checked_json(Path(__file__).with_name("compatibility-plan.json"), EXPECTED_PLAN)
|
||||
manifest = checked_json(Path(__file__).with_name("model-manifest.json"), EXPECTED_MANIFEST)
|
||||
start = time.perf_counter()
|
||||
for item in manifest["files"]:
|
||||
path = DESTINATION / item["path"]
|
||||
if path.stat().st_size != item["bytes"] or digest(path) != item["sha256"]:
|
||||
raise ValueError("Pinned diagnostic asset mismatch")
|
||||
verification_ms = (time.perf_counter() - start) * 1000
|
||||
start = time.perf_counter()
|
||||
tokenizer = Tokenizer.from_file(str(DESTINATION / "tokenizer.json"))
|
||||
special = tokenizer.num_special_tokens_to_add(True)
|
||||
encodings = []
|
||||
for query, passage in plan["pairs"]:
|
||||
query_encoding = tokenizer.encode(query, add_special_tokens=False)
|
||||
query_encoding.truncate(plan["max_query_tokens"])
|
||||
passage_encoding = tokenizer.encode(passage, add_special_tokens=False)
|
||||
passage_encoding.truncate(plan["max_pair_tokens"] - len(query_encoding.ids) - special)
|
||||
encoding = tokenizer.post_process(query_encoding, passage_encoding, add_special_tokens=True)
|
||||
if len(encoding.ids) > plan["max_pair_tokens"]:
|
||||
raise ValueError("Pair token budget exceeded")
|
||||
encodings.append(encoding)
|
||||
tokenization_ms = (time.perf_counter() - start) * 1000
|
||||
options = ort.SessionOptions()
|
||||
options.intra_op_num_threads = 1
|
||||
options.inter_op_num_threads = 1
|
||||
options.execution_mode = ort.ExecutionMode.ORT_SEQUENTIAL
|
||||
start = time.perf_counter()
|
||||
session = ort.InferenceSession(str(DESTINATION / "onnx/model_quint8_avx2.onnx"), sess_options=options, providers=["CPUExecutionProvider"])
|
||||
startup_ms = (time.perf_counter() - start) * 1000
|
||||
if session.get_providers() != ["CPUExecutionProvider"]:
|
||||
raise ValueError("Unexpected execution provider")
|
||||
names = [item.name for item in session.get_inputs()]
|
||||
if not set(names) <= {"input_ids", "attention_mask", "token_type_ids"}:
|
||||
raise ValueError("Unexpected model input contract")
|
||||
def score(items):
|
||||
width = max(len(item.ids) for item in items)
|
||||
ids = np.full((len(items), width), 1, dtype=np.int64)
|
||||
mask = np.zeros_like(ids)
|
||||
types = np.zeros_like(ids)
|
||||
for index, item in enumerate(items):
|
||||
count = len(item.ids)
|
||||
ids[index, :count] = item.ids
|
||||
mask[index, :count] = item.attention_mask
|
||||
types[index, :count] = item.type_ids
|
||||
values = {"input_ids": ids, "attention_mask": mask, "token_type_ids": types}
|
||||
logits = session.run(None, {name: values[name] for name in names})[0]
|
||||
if logits.shape != (len(items), 1) or not np.isfinite(logits).all():
|
||||
raise ValueError("Invalid classifier output")
|
||||
return logits[:, 0]
|
||||
start = time.perf_counter()
|
||||
serial = np.asarray([float(score([item])[0]) for item in encodings])
|
||||
first_serial_ms = (time.perf_counter() - start) * 1000
|
||||
start = time.perf_counter()
|
||||
batch = score(encodings)
|
||||
first_batch_ms = (time.perf_counter() - start) * 1000
|
||||
parity = bool(np.allclose(serial, batch, atol=plan["parity_absolute_tolerance"], rtol=plan["parity_relative_tolerance"]))
|
||||
serial_timings = []
|
||||
batch_timings = []
|
||||
for _ in range(plan["warm_iterations"]):
|
||||
start = time.perf_counter()
|
||||
for item in encodings:
|
||||
score([item])
|
||||
serial_timings.append((time.perf_counter() - start) * 1000)
|
||||
start = time.perf_counter()
|
||||
score(encodings)
|
||||
batch_timings.append((time.perf_counter() - start) * 1000)
|
||||
def timing(values):
|
||||
return {"median_ms": statistics.median(values), "p95_ms": sorted(values)[math.ceil(len(values) * .95) - 1], "samples_ms": values}
|
||||
report = {"complete": True, "plan_sha256": EXPECTED_PLAN, "manifest_sha256": EXPECTED_MANIFEST,
|
||||
"platform": platform.platform(), "python": platform.python_version(), "onnxruntime": ort.__version__,
|
||||
"numpy": np.__version__, "providers": session.get_providers(), "cpu_threads": 1,
|
||||
"asset_verification_ms": verification_ms, "tokenization_ms": tokenization_ms, "session_startup_ms": startup_ms,
|
||||
"pair_tokens": [len(item.ids) for item in encodings], "first_serial_ms": first_serial_ms, "first_batch_ms": first_batch_ms,
|
||||
"serial_logits": serial.tolist(), "batch_logits": batch.tolist(), "parity_passed": parity,
|
||||
"max_absolute_difference": float(np.max(np.abs(serial - batch))),
|
||||
"warm_serial": timing(serial_timings), "warm_batch": timing(batch_timings),
|
||||
"limitations": "Five synthetic short pairs; classifier compatibility/cost only. No search pipeline or absence calibration. No confidence probability or suppression."}
|
||||
output.parent.mkdir(parents=True, exist_ok=True)
|
||||
with tempfile.NamedTemporaryFile(mode="w", encoding="utf-8", dir=output.parent, delete=False) as handle:
|
||||
temporary = Path(handle.name)
|
||||
json.dump(report, handle, indent=2)
|
||||
handle.write("\n")
|
||||
try:
|
||||
os.link(temporary, output)
|
||||
finally:
|
||||
temporary.unlink(missing_ok=True)
|
||||
print(json.dumps({"output": str(output), "parity_passed": parity, "max_absolute_difference": report["max_absolute_difference"], "warm_serial": report["warm_serial"]["p95_ms"], "warm_batch": report["warm_batch"]["p95_ms"]}), flush=True)
|
||||
if not parity:
|
||||
raise SystemExit("Serial/batch parity failed; do not deploy")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,53 @@
|
||||
"""Download only pinned diagnostic assets into owned ignored data."""
|
||||
import argparse
|
||||
import hashlib
|
||||
import json
|
||||
from pathlib import Path
|
||||
import time
|
||||
import urllib.request
|
||||
import uuid
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[2]
|
||||
DESTINATION = ROOT / ".searchi" / "rerank-eval" / "model"
|
||||
|
||||
|
||||
def digest(path):
|
||||
checksum = hashlib.sha256()
|
||||
with path.open("rb") as handle:
|
||||
for chunk in iter(lambda: handle.read(65536), b""):
|
||||
checksum.update(chunk)
|
||||
return checksum.hexdigest()
|
||||
|
||||
|
||||
def main():
|
||||
parser=argparse.ArgumentParser()
|
||||
parser.add_argument("--reference",action="store_true")
|
||||
args=parser.parse_args()
|
||||
name="reference-manifest.json" if args.reference else "model-manifest.json"
|
||||
manifest = json.loads(Path(__file__).with_name(name).read_text(encoding="utf-8"))
|
||||
for item in manifest["files"]:
|
||||
destination = DESTINATION / item["path"]
|
||||
if destination.is_file() and destination.stat().st_size == item["bytes"] and digest(destination) == item["sha256"]:
|
||||
print("verified cache", item["path"], flush=True)
|
||||
continue
|
||||
destination.parent.mkdir(parents=True, exist_ok=True)
|
||||
temporary = destination.with_name(destination.name + "." + uuid.uuid4().hex + ".partial")
|
||||
total = 0
|
||||
start = time.monotonic()
|
||||
try:
|
||||
with urllib.request.urlopen(item["url"], timeout=20) as response, temporary.open("xb") as output:
|
||||
while chunk := response.read(65536):
|
||||
total += len(chunk)
|
||||
if total > item["bytes"] or time.monotonic() - start > 180:
|
||||
raise ValueError("Download exceeded size/deadline")
|
||||
output.write(chunk)
|
||||
if total != item["bytes"] or digest(temporary) != item["sha256"]:
|
||||
raise ValueError("Pinned asset hash/size mismatch")
|
||||
temporary.replace(destination)
|
||||
print("verified download", item["path"], total, flush=True)
|
||||
finally:
|
||||
temporary.unlink(missing_ok=True)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,50 @@
|
||||
{
|
||||
"model_id": "cross-encoder/mmarco-mMiniLMv2-L12-H384-v1",
|
||||
"revision": "1427fd652930e4ba29e8149678df786c240d8825",
|
||||
"purpose": "diagnostic text reranker; Google EmbeddingGemma2 remains retrieval",
|
||||
"license": "Apache-2.0 per pinned publisher card; text from Apache",
|
||||
"files": [
|
||||
{
|
||||
"path": "onnx/model_quint8_avx2.onnx",
|
||||
"bytes": 118620016,
|
||||
"sha256": "6c2513767fb63d008a4377bef7a7a3555433d9436342bb53e35a3a72ffc52d4b",
|
||||
"url": "https://huggingface.co/cross-encoder/mmarco-mMiniLMv2-L12-H384-v1/resolve/1427fd652930e4ba29e8149678df786c240d8825/onnx/model_quint8_avx2.onnx"
|
||||
},
|
||||
{
|
||||
"path": "tokenizer.json",
|
||||
"bytes": 17082660,
|
||||
"sha256": "62c24cdc13d4c9952d63718d6c9fa4c287974249e16b7ade6d5a85e7bbb75626",
|
||||
"url": "https://huggingface.co/cross-encoder/mmarco-mMiniLMv2-L12-H384-v1/resolve/1427fd652930e4ba29e8149678df786c240d8825/tokenizer.json"
|
||||
},
|
||||
{
|
||||
"path": "config.json",
|
||||
"bytes": 891,
|
||||
"sha256": "cc2cfe51aa3fd759d21d21acf5dfd6994aa67a3c9210636d22e143699d336c77",
|
||||
"url": "https://huggingface.co/cross-encoder/mmarco-mMiniLMv2-L12-H384-v1/resolve/1427fd652930e4ba29e8149678df786c240d8825/config.json"
|
||||
},
|
||||
{
|
||||
"path": "tokenizer_config.json",
|
||||
"bytes": 435,
|
||||
"sha256": "e7fbfbfa6347b4e414c1cee50d142e2c2f9a895dad68b068ae83a8b564c3837e",
|
||||
"url": "https://huggingface.co/cross-encoder/mmarco-mMiniLMv2-L12-H384-v1/resolve/1427fd652930e4ba29e8149678df786c240d8825/tokenizer_config.json"
|
||||
},
|
||||
{
|
||||
"path": "special_tokens_map.json",
|
||||
"bytes": 239,
|
||||
"sha256": "378eb3bf733eb16e65792d7e3fda5b8a4631387ca04d2015199c4d4f22ae554d",
|
||||
"url": "https://huggingface.co/cross-encoder/mmarco-mMiniLMv2-L12-H384-v1/resolve/1427fd652930e4ba29e8149678df786c240d8825/special_tokens_map.json"
|
||||
},
|
||||
{
|
||||
"path": "README.md",
|
||||
"bytes": 2278,
|
||||
"sha256": "474736a65d6393a060119a8dc304563af67af4d8d86ccfee4a05dd0df107fc11",
|
||||
"url": "https://huggingface.co/cross-encoder/mmarco-mMiniLMv2-L12-H384-v1/resolve/1427fd652930e4ba29e8149678df786c240d8825/README.md"
|
||||
},
|
||||
{
|
||||
"path": "LICENSE-APACHE-2.0.txt",
|
||||
"bytes": 11358,
|
||||
"sha256": "cfc7749b96f63bd31c3c42b5c471bf756814053e847c10f3eb003417bc523d30",
|
||||
"url": "https://www.apache.org/licenses/LICENSE-2.0.txt"
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,56 @@
|
||||
{
|
||||
"model_id": "cross-encoder/mmarco-mMiniLMv2-L12-H384-v1",
|
||||
"revision": "1427fd652930e4ba29e8149678df786c240d8825",
|
||||
"purpose": "diagnostic text reranker; Google EmbeddingGemma2 remains retrieval",
|
||||
"license": "Apache-2.0 per pinned publisher card; text from Apache",
|
||||
"files": [
|
||||
{
|
||||
"path": "onnx/model_quint8_avx2.onnx",
|
||||
"bytes": 118620016,
|
||||
"sha256": "6c2513767fb63d008a4377bef7a7a3555433d9436342bb53e35a3a72ffc52d4b",
|
||||
"url": "https://huggingface.co/cross-encoder/mmarco-mMiniLMv2-L12-H384-v1/resolve/1427fd652930e4ba29e8149678df786c240d8825/onnx/model_quint8_avx2.onnx"
|
||||
},
|
||||
{
|
||||
"path": "tokenizer.json",
|
||||
"bytes": 17082660,
|
||||
"sha256": "62c24cdc13d4c9952d63718d6c9fa4c287974249e16b7ade6d5a85e7bbb75626",
|
||||
"url": "https://huggingface.co/cross-encoder/mmarco-mMiniLMv2-L12-H384-v1/resolve/1427fd652930e4ba29e8149678df786c240d8825/tokenizer.json"
|
||||
},
|
||||
{
|
||||
"path": "config.json",
|
||||
"bytes": 891,
|
||||
"sha256": "cc2cfe51aa3fd759d21d21acf5dfd6994aa67a3c9210636d22e143699d336c77",
|
||||
"url": "https://huggingface.co/cross-encoder/mmarco-mMiniLMv2-L12-H384-v1/resolve/1427fd652930e4ba29e8149678df786c240d8825/config.json"
|
||||
},
|
||||
{
|
||||
"path": "tokenizer_config.json",
|
||||
"bytes": 435,
|
||||
"sha256": "e7fbfbfa6347b4e414c1cee50d142e2c2f9a895dad68b068ae83a8b564c3837e",
|
||||
"url": "https://huggingface.co/cross-encoder/mmarco-mMiniLMv2-L12-H384-v1/resolve/1427fd652930e4ba29e8149678df786c240d8825/tokenizer_config.json"
|
||||
},
|
||||
{
|
||||
"path": "special_tokens_map.json",
|
||||
"bytes": 239,
|
||||
"sha256": "378eb3bf733eb16e65792d7e3fda5b8a4631387ca04d2015199c4d4f22ae554d",
|
||||
"url": "https://huggingface.co/cross-encoder/mmarco-mMiniLMv2-L12-H384-v1/resolve/1427fd652930e4ba29e8149678df786c240d8825/special_tokens_map.json"
|
||||
},
|
||||
{
|
||||
"path": "README.md",
|
||||
"bytes": 2278,
|
||||
"sha256": "474736a65d6393a060119a8dc304563af67af4d8d86ccfee4a05dd0df107fc11",
|
||||
"url": "https://huggingface.co/cross-encoder/mmarco-mMiniLMv2-L12-H384-v1/resolve/1427fd652930e4ba29e8149678df786c240d8825/README.md"
|
||||
},
|
||||
{
|
||||
"path": "LICENSE-APACHE-2.0.txt",
|
||||
"bytes": 11358,
|
||||
"sha256": "cfc7749b96f63bd31c3c42b5c471bf756814053e847c10f3eb003417bc523d30",
|
||||
"url": "https://www.apache.org/licenses/LICENSE-2.0.txt"
|
||||
},
|
||||
{
|
||||
"path": "onnx/model.onnx",
|
||||
"bytes": 470883696,
|
||||
"sha256": "3e9a03ed1e966f7c5288dd4230e3d6a9bf5e3a170a06f1f4241c5bca12c6487c",
|
||||
"url": "https://huggingface.co/cross-encoder/mmarco-mMiniLMv2-L12-H384-v1/resolve/1427fd652930e4ba29e8149678df786c240d8825/onnx/model.onnx"
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,44 @@
|
||||
{
|
||||
"version": 2,
|
||||
"status": "pre-inference F32 reference parity and 256-token cost contract; original INT8 failure unchanged",
|
||||
"cpu_provider": "CPUExecutionProvider",
|
||||
"intra_op_threads": 1,
|
||||
"inter_op_threads": 1,
|
||||
"parallel_execution": false,
|
||||
"max_pairs": 6,
|
||||
"max_pair_tokens": 256,
|
||||
"max_query_tokens": 64,
|
||||
"warm_iterations": 20,
|
||||
"parity_absolute_tolerance": 0.001,
|
||||
"parity_relative_tolerance": 0.001,
|
||||
"added_warm_p95_target_ms": 200,
|
||||
"no_suppression": true,
|
||||
"pairs": [
|
||||
[
|
||||
"How do we cancel the supplier partnership?",
|
||||
"Either party may terminate the supplier contract by giving sixty days written notice."
|
||||
],
|
||||
[
|
||||
"Wie beenden wir die Zusammenarbeit mit dem Lieferanten?",
|
||||
"Jede Partei kann die Zusammenarbeit mit dem Lieferanten mit einer Frist von sechzig Tagen kündigen."
|
||||
],
|
||||
[
|
||||
"How do we cancel the supplier partnership?",
|
||||
"Tomatoes require sunshine and consistent watering."
|
||||
],
|
||||
[
|
||||
"Wie beenden wir die Zusammenarbeit mit dem Lieferanten?",
|
||||
"Der Garten benötigt Sonnenlicht und regelmäßige Bewässerung."
|
||||
],
|
||||
[
|
||||
"Where does the code implement retry backoff?",
|
||||
"fn retry_delay(attempt: u32) -> u64 { 100 * (1 << attempt.min(6)) }"
|
||||
],
|
||||
[
|
||||
"supplier partnership cancellation notice supplier partnership cancellation notice supplier partnership cancellation notice supplier partnership cancellation notice supplier partnership cancellation notice supplier partnership cancellation notice supplier partnership cancellation notice supplier partnership cancellation notice supplier partnership cancellation notice supplier partnership cancellation notice supplier partnership cancellation notice supplier partnership cancellation notice supplier partnership cancellation notice supplier partnership cancellation notice supplier partnership cancellation notice supplier partnership cancellation notice supplier partnership cancellation notice supplier partnership cancellation notice supplier partnership cancellation notice supplier partnership cancellation notice supplier partnership cancellation notice supplier partnership cancellation notice supplier partnership cancellation notice supplier partnership cancellation notice supplier partnership cancellation notice supplier partnership cancellation notice supplier partnership cancellation notice supplier partnership cancellation notice supplier partnership cancellation notice supplier partnership cancellation notice ",
|
||||
"Either party may terminate the supplier partnership with sixty days written notice. Either party may terminate the supplier partnership with sixty days written notice. Either party may terminate the supplier partnership with sixty days written notice. Either party may terminate the supplier partnership with sixty days written notice. Either party may terminate the supplier partnership with sixty days written notice. Either party may terminate the supplier partnership with sixty days written notice. Either party may terminate the supplier partnership with sixty days written notice. Either party may terminate the supplier partnership with sixty days written notice. Either party may terminate the supplier partnership with sixty days written notice. Either party may terminate the supplier partnership with sixty days written notice. Either party may terminate the supplier partnership with sixty days written notice. Either party may terminate the supplier partnership with sixty days written notice. Either party may terminate the supplier partnership with sixty days written notice. Either party may terminate the supplier partnership with sixty days written notice. Either party may terminate the supplier partnership with sixty days written notice. Either party may terminate the supplier partnership with sixty days written notice. Either party may terminate the supplier partnership with sixty days written notice. Either party may terminate the supplier partnership with sixty days written notice. Either party may terminate the supplier partnership with sixty days written notice. Either party may terminate the supplier partnership with sixty days written notice. Either party may terminate the supplier partnership with sixty days written notice. Either party may terminate the supplier partnership with sixty days written notice. Either party may terminate the supplier partnership with sixty days written notice. Either party may terminate the supplier partnership with sixty days written notice. Either party may terminate the supplier partnership with sixty days written notice. Either party may terminate the supplier partnership with sixty days written notice. Either party may terminate the supplier partnership with sixty days written notice. Either party may terminate the supplier partnership with sixty days written notice. Either party may terminate the supplier partnership with sixty days written notice. Either party may terminate the supplier partnership with sixty days written notice. Either party may terminate the supplier partnership with sixty days written notice. Either party may terminate the supplier partnership with sixty days written notice. Either party may terminate the supplier partnership with sixty days written notice. Either party may terminate the supplier partnership with sixty days written notice. Either party may terminate the supplier partnership with sixty days written notice. Either party may terminate the supplier partnership with sixty days written notice. Either party may terminate the supplier partnership with sixty days written notice. Either party may terminate the supplier partnership with sixty days written notice. Either party may terminate the supplier partnership with sixty days written notice. Either party may terminate the supplier partnership with sixty days written notice. Either party may terminate the supplier partnership with sixty days written notice. Either party may terminate the supplier partnership with sixty days written notice. Either party may terminate the supplier partnership with sixty days written notice. Either party may terminate the supplier partnership with sixty days written notice. Either party may terminate the supplier partnership with sixty days written notice. Either party may terminate the supplier partnership with sixty days written notice. Either party may terminate the supplier partnership with sixty days written notice. Either party may terminate the supplier partnership with sixty days written notice. Either party may terminate the supplier partnership with sixty days written notice. Either party may terminate the supplier partnership with sixty days written notice. Either party may terminate the supplier partnership with sixty days written notice. Either party may terminate the supplier partnership with sixty days written notice. Either party may terminate the supplier partnership with sixty days written notice. Either party may terminate the supplier partnership with sixty days written notice. Either party may terminate the supplier partnership with sixty days written notice. Either party may terminate the supplier partnership with sixty days written notice. Either party may terminate the supplier partnership with sixty days written notice. Either party may terminate the supplier partnership with sixty days written notice. Either party may terminate the supplier partnership with sixty days written notice. Either party may terminate the supplLine truncated
|
||||
]
|
||||
],
|
||||
"reference_model": "onnx/model.onnx",
|
||||
"comparison_model": "onnx/model_quint8_avx2.onnx"
|
||||
}
|
||||
@@ -0,0 +1,143 @@
|
||||
"""Owned CPU-only cross-encoder compatibility/cost diagnostic; never alters search."""
|
||||
import argparse
|
||||
import hashlib
|
||||
import json
|
||||
import math
|
||||
import os
|
||||
from pathlib import Path
|
||||
import platform
|
||||
import statistics
|
||||
import tempfile
|
||||
import time
|
||||
|
||||
import numpy as np
|
||||
import onnxruntime as ort
|
||||
from tokenizers import Tokenizer
|
||||
from download import DESTINATION, digest
|
||||
|
||||
EXPECTED_PLAN = "174867d4212b222e5302b1b16832b371c6bd38ec991f4145c952ed35446341f1"
|
||||
EXPECTED_MANIFEST = "3c252dfc10e03ebba59a7ff0e33e6e2183a26c71862c52f0700f27ba659142be"
|
||||
|
||||
|
||||
def checked_json(path, expected):
|
||||
data = path.read_bytes()
|
||||
if hashlib.sha256(data).hexdigest() != expected:
|
||||
raise ValueError("Frozen diagnostic inputs changed")
|
||||
return json.loads(data)
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("--output", type=Path, required=True)
|
||||
args = parser.parse_args()
|
||||
owned_root = DESTINATION.parents[1].resolve()
|
||||
output = args.output.resolve()
|
||||
if not output.is_relative_to(owned_root) or output.exists():
|
||||
raise ValueError("Output must be new and under owned .searchi data")
|
||||
if os.environ.get("SEARCHI_FORCE_CPU") != "1":
|
||||
raise ValueError("Windows VM diagnostic requires explicit SEARCHI_FORCE_CPU=1")
|
||||
plan = checked_json(Path(__file__).with_name("reference-plan.json"), EXPECTED_PLAN)
|
||||
manifest = checked_json(Path(__file__).with_name("reference-manifest.json"), EXPECTED_MANIFEST)
|
||||
start = time.perf_counter()
|
||||
for item in manifest["files"]:
|
||||
path = DESTINATION / item["path"]
|
||||
if path.stat().st_size != item["bytes"] or digest(path) != item["sha256"]:
|
||||
raise ValueError("Pinned diagnostic asset mismatch")
|
||||
verification_ms = (time.perf_counter() - start) * 1000
|
||||
start = time.perf_counter()
|
||||
tokenizer = Tokenizer.from_file(str(DESTINATION / "tokenizer.json"))
|
||||
special = tokenizer.num_special_tokens_to_add(True)
|
||||
encodings = []
|
||||
for query, passage in plan["pairs"]:
|
||||
query_encoding = tokenizer.encode(query, add_special_tokens=False)
|
||||
query_encoding.truncate(plan["max_query_tokens"])
|
||||
passage_encoding = tokenizer.encode(passage, add_special_tokens=False)
|
||||
passage_encoding.truncate(plan["max_pair_tokens"] - len(query_encoding.ids) - special)
|
||||
encoding = tokenizer.post_process(query_encoding, passage_encoding, add_special_tokens=True)
|
||||
if len(encoding.ids) > plan["max_pair_tokens"]:
|
||||
raise ValueError("Pair token budget exceeded")
|
||||
encodings.append(encoding)
|
||||
tokenization_ms = (time.perf_counter() - start) * 1000
|
||||
options = ort.SessionOptions()
|
||||
options.intra_op_num_threads = 1
|
||||
options.inter_op_num_threads = 1
|
||||
options.execution_mode = ort.ExecutionMode.ORT_SEQUENTIAL
|
||||
start = time.perf_counter()
|
||||
session = ort.InferenceSession(str(DESTINATION / plan["reference_model"]), sess_options=options, providers=["CPUExecutionProvider"])
|
||||
startup_ms = (time.perf_counter() - start) * 1000
|
||||
if session.get_providers() != ["CPUExecutionProvider"]:
|
||||
raise ValueError("Unexpected execution provider")
|
||||
names = [item.name for item in session.get_inputs()]
|
||||
if not set(names) <= {"input_ids", "attention_mask", "token_type_ids"}:
|
||||
raise ValueError("Unexpected model input contract")
|
||||
def score(items):
|
||||
width = max(len(item.ids) for item in items)
|
||||
ids = np.full((len(items), width), 1, dtype=np.int64)
|
||||
mask = np.zeros_like(ids)
|
||||
types = np.zeros_like(ids)
|
||||
for index, item in enumerate(items):
|
||||
count = len(item.ids)
|
||||
ids[index, :count] = item.ids
|
||||
mask[index, :count] = item.attention_mask
|
||||
types[index, :count] = item.type_ids
|
||||
values = {"input_ids": ids, "attention_mask": mask, "token_type_ids": types}
|
||||
logits = session.run(None, {name: values[name] for name in names})[0]
|
||||
if logits.shape != (len(items), 1) or not np.isfinite(logits).all():
|
||||
raise ValueError("Invalid classifier output")
|
||||
return logits[:, 0]
|
||||
start = time.perf_counter()
|
||||
serial = np.asarray([float(score([item])[0]) for item in encodings])
|
||||
first_serial_ms = (time.perf_counter() - start) * 1000
|
||||
start = time.perf_counter()
|
||||
batch = score(encodings)
|
||||
first_batch_ms = (time.perf_counter() - start) * 1000
|
||||
parity = bool(np.allclose(serial, batch, atol=plan["parity_absolute_tolerance"], rtol=plan["parity_relative_tolerance"]))
|
||||
serial_timings = []
|
||||
batch_timings = []
|
||||
for _ in range(plan["warm_iterations"]):
|
||||
start = time.perf_counter()
|
||||
for item in encodings:
|
||||
score([item])
|
||||
serial_timings.append((time.perf_counter() - start) * 1000)
|
||||
start = time.perf_counter()
|
||||
score(encodings)
|
||||
batch_timings.append((time.perf_counter() - start) * 1000)
|
||||
reference_session = session
|
||||
session = ort.InferenceSession(str(DESTINATION / plan["comparison_model"]), sess_options=options, providers=["CPUExecutionProvider"])
|
||||
if session.get_providers() != ["CPUExecutionProvider"]:
|
||||
raise ValueError("Unexpected comparison provider")
|
||||
quantized_serial = np.asarray([float(score([item])[0]) for item in encodings])
|
||||
quantized_batch = score(encodings)
|
||||
session = reference_session
|
||||
def timing(values):
|
||||
return {"median_ms": statistics.median(values), "p95_ms": sorted(values)[math.ceil(len(values) * .95) - 1], "samples_ms": values}
|
||||
report = {"complete": True, "plan_sha256": EXPECTED_PLAN, "manifest_sha256": EXPECTED_MANIFEST,
|
||||
"platform": platform.platform(), "python": platform.python_version(), "onnxruntime": ort.__version__,
|
||||
"numpy": np.__version__, "providers": session.get_providers(), "cpu_threads": 1,
|
||||
"asset_verification_ms": verification_ms, "tokenization_ms": tokenization_ms, "session_startup_ms": startup_ms,
|
||||
"pair_tokens": [len(item.ids) for item in encodings], "first_serial_ms": first_serial_ms, "first_batch_ms": first_batch_ms,
|
||||
"reference_model": plan["reference_model"], "comparison_model": plan["comparison_model"],
|
||||
"serial_logits": serial.tolist(), "batch_logits": batch.tolist(), "parity_passed": parity,
|
||||
"quantized_serial_logits": quantized_serial.tolist(), "quantized_batch_logits": quantized_batch.tolist(),
|
||||
"quantized_vs_reference_serial_max_abs": float(np.max(np.abs(serial-quantized_serial))),
|
||||
"quantized_batch_max_abs": float(np.max(np.abs(quantized_serial-quantized_batch))),
|
||||
"five_short_pairs_ordering_agrees": bool(np.array_equal(np.argsort(serial[:5]),np.argsort(quantized_serial[:5]))),
|
||||
"max_absolute_difference": float(np.max(np.abs(serial - batch))),
|
||||
"warm_serial": timing(serial_timings), "warm_batch": timing(batch_timings),
|
||||
"limitations": "Five synthetic short pairs plus one 256-token cost pair; classifier compatibility/cost only. No search pipeline or absence calibration. No confidence probability or suppression."}
|
||||
output.parent.mkdir(parents=True, exist_ok=True)
|
||||
with tempfile.NamedTemporaryFile(mode="w", encoding="utf-8", dir=output.parent, delete=False) as handle:
|
||||
temporary = Path(handle.name)
|
||||
json.dump(report, handle, indent=2)
|
||||
handle.write("\n")
|
||||
try:
|
||||
os.link(temporary, output)
|
||||
finally:
|
||||
temporary.unlink(missing_ok=True)
|
||||
print(json.dumps({"output": str(output), "parity_passed": parity, "max_absolute_difference": report["max_absolute_difference"], "warm_serial": report["warm_serial"]["p95_ms"], "warm_batch": report["warm_batch"]["p95_ms"]}), flush=True)
|
||||
if not parity:
|
||||
raise SystemExit("Serial/batch parity failed; do not deploy")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
Reference in new issue
Block a user