karl
caf6a2f82e
Core CI / core (ubuntu-latest) (push) Failing after 3m27s
Core CI / core (macos-latest) (push) Canceled after 0s
Core CI / core (windows-latest) (push) Canceled after 0s
Core CI / desktop (ubuntu-latest) (push) Successful in 5m45s
Core CI / desktop (macos-latest) (push) Canceled after 0s
Core CI / desktop (windows-latest) (push) Canceled after 0s
161 lines
8.5 KiB
Markdown
161 lines
8.5 KiB
Markdown
# CPU semantic inference proof
|
|
|
|
**Historical experiment, not the accepted Searchi backend.** The user requires
|
|
Google EmbeddingGemma 2 as used by DigUp. MiniLM results below establish only
|
|
runtime plumbing and must not be presented as completed product semantic search.
|
|
The implementation is being corrected to use the required Google model.
|
|
|
|
This milestone runs genuine neural embeddings locally. It does not yet connect
|
|
semantic vectors to the SQLite search index or desktop interface. Production
|
|
retrieval quality, large-corpus performance, and Linux/macOS runtime packaging
|
|
remain unverified.
|
|
|
|
## Model and runtime decision
|
|
|
|
The first working baseline is the quantized ONNX export of
|
|
`sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2`, published by
|
|
`Xenova`. The upstream model card describes 384-dimensional text embeddings,
|
|
attention-mask mean pooling, a 128-token maximum, and Apache-2.0 licensing.
|
|
It covers multiple languages. This is a text-only baseline; it does not claim
|
|
EmbeddingGemma 2's multimodal features or compatibility.
|
|
|
|
Sources reviewed on 2026-10-12:
|
|
|
|
- [Original model card and pooling architecture](https://huggingface.co/sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2)
|
|
- [ONNX export publisher](https://huggingface.co/Xenova/paraphrase-multilingual-MiniLM-L12-v2)
|
|
- [ONNX Runtime Python session API](https://onnxruntime.ai/docs/api/python/api_summary.html)
|
|
- [Rust ort dynamic-library initialization](https://docs.rs/ort/2.0.0-rc.13/ort/environment/fn.init_from.html)
|
|
- [Rust ort session configuration](https://docs.rs/ort/2.0.0-rc.13/ort/session/builder/struct.SessionBuilder.html)
|
|
|
|
Inference explicitly selects `CPUExecutionProvider`. Python uses the CPU
|
|
`onnxruntime` wheel, never `onnxruntime-gpu`. The native Rust crate disables ort
|
|
default features: there are no binary downloads, GPU provider features, or model
|
|
HTTP client features. It loads an explicitly supplied local runtime library and
|
|
disables runtime telemetry. Tokenization is sequential; ONNX sessions use
|
|
sequential graph execution with bounded intra-op threads and one inter-op thread.
|
|
|
|
## Reproduce on Windows
|
|
|
|
Run these commands from the repository root. Provisioning commands intentionally
|
|
use the network; the inference command does not provision anything.
|
|
|
|
```powershell
|
|
python -m venv tools/semantic-proof/.venv
|
|
& tools/semantic-proof/.venv/Scripts/python.exe -m pip install --only-binary=:all: -r tools/semantic-proof/requirements.txt
|
|
& tools/semantic-proof/.venv/Scripts/python.exe tools/semantic-proof/download.py
|
|
& tools/semantic-proof/.venv/Scripts/python.exe tools/semantic-proof/proof.py --threads 1 --output tools/semantic-proof/windows-cpu-result.json
|
|
```
|
|
|
|
On Linux/macOS, use `.venv/bin/python` instead of `.venv/Scripts/python.exe`.
|
|
These instructions have only been executed on Windows. Wheel availability and
|
|
native library distribution must be checked on other platforms.
|
|
|
|
`download.py` downloads exactly two revision-pinned artifacts, streams them to a
|
|
temporary file, rejects oversized downloads, checks byte length and SHA256, then
|
|
atomically replaces the local asset. Assets occupy approximately 136 MB in
|
|
decimal units and are ignored by Git. Inference validates both files again and
|
|
fails clearly when either is missing or damaged. It never contacts the model
|
|
hub. The Python proof sets offline environment flags and rejects Python socket
|
|
connect, name-resolution, and bind audit events throughout import and inference.
|
|
This guard checks Python socket activity; it is not an operating-system firewall
|
|
against arbitrary native libraries.
|
|
|
|
The proof intentionally exits nonzero if any fixture query misses top-1. Its
|
|
JSON report is written before the quality gate, preserving failures for review.
|
|
|
|
## Pinned artifacts and licensing
|
|
|
|
Revision: `2c4055b12046f11709e9df2c122e59ffbdc2f900`.
|
|
The manifest in `tools/semantic-proof/model-manifest.json` records the same
|
|
model contract and hashes used by the native adapter.
|
|
|
|
| Artifact | Bytes | SHA256 |
|
|
| --- | ---: | --- |
|
|
| `onnx/model_quantized.onnx` | 118,308,126 | `66fc00f5f29afcaff34092e1bdd20008ca3918265a82fb9695a551e510cc4ebc` |
|
|
| `tokenizer.json` | 17,082,913 | `b60b6b43406a48bf3638526314f3d232d97058bc93472ff2de930d43686fa441` |
|
|
|
|
The base model card specifies Apache-2.0. The export card identifies that base
|
|
model but does not display its own license declaration. Before redistributing
|
|
the ONNX export in installers, confirm export licensing and preserve the base
|
|
license, attribution, and applicable notices. Current code explicitly downloads
|
|
user-local assets and does not ship model weights or runtime DLLs in Git.
|
|
ONNX Runtime uses MIT licensing; ort uses MIT or Apache-2.0 and tokenizers uses
|
|
Apache-2.0. Release packaging must retain third-party notices and validate
|
|
platform-specific runtime hashes independently; a model hash does not attest
|
|
to the runtime library.
|
|
|
|
The tested Windows ONNX Runtime 1.31.0 DLL is 19,288,888 bytes with SHA256
|
|
`90497e3d406c09f4d59d364c696a2f93dc02f6ef13741b3b89eec95bdb4a3806`.
|
|
The adapter currently trusts the explicitly supplied runtime path; it validates
|
|
model hashes, not runtime-library hashes. A release distribution must add a
|
|
platform runtime manifest before shipping. `requirements-lock-windows.txt`
|
|
captures the exact Python environment used for this proof.
|
|
|
|
## Measured Windows VM result
|
|
|
|
`tools/semantic-proof/windows-cpu-result.json` contains the full measurement and
|
|
rankings. The run used Windows 11 AMD64, Python 3.14.8, ONNX Runtime 1.31.0,
|
|
four visible logical CPUs, and **one inference thread with only the CPU provider**.
|
|
|
|
| Metric | Measurement |
|
|
| --- | ---: |
|
|
| Fixture documents / queries | 8 / 6 |
|
|
| Timed warm query runs | 60 |
|
|
| Artifact verification plus runtime/model load | approximately 3.4 seconds |
|
|
| Corpus batch embedding | approximately 145 ms |
|
|
| Warm query p50 / p95 | approximately 3.1 / 11.9 ms |
|
|
| Top-1 accuracy | 5/6 (83.3%) |
|
|
| Recall at 3 | 6/6 (100%) |
|
|
|
|
The corpus includes English and German documents, same-language semantic
|
|
queries, and cross-language queries. A question about a postponed product
|
|
release ranks the supplier contract above the expected meeting notes. That
|
|
failure remains in the fixture and blocks the strict top-1 quality gate.
|
|
This small demonstration supports the feasibility of CPU inference; it does
|
|
not establish large-corpus latency, robust multilingual quality, or production
|
|
readiness. Quantization and batch shape can affect similarity scores.
|
|
|
|
## Native integration contract
|
|
|
|
`searchi-semantic` exposes:
|
|
|
|
```rust
|
|
CpuEmbedder::load(model_dir, runtime_library, threads) -> Result<CpuEmbedder>
|
|
CpuEmbedder::encode(&mut self, texts: &[&str]) -> Result<Vec<Vec<f32>>>
|
|
DIMENSIONS // 384
|
|
MODEL_FINGERPRINT // revision + quantization + pooling + normalization + token cap
|
|
```
|
|
|
|
Inputs are bounded to eight texts per batch, 64 KiB UTF-8 per text, and 128
|
|
tokens. Outputs are finite, normalized `f32` vectors. The runtime library is
|
|
process-global; adapter initialization rejects attempts to switch libraries or
|
|
silently accept an environment initialized outside the adapter. Supply a known,
|
|
trusted runtime before calling any other ort APIs in the process.
|
|
|
|
Default unit tests need no model or runtime. The real-artifact test is opt-in:
|
|
|
|
```powershell
|
|
$env:SEARCHI_MODEL_DIR = (Resolve-Path tools/semantic-proof/model).Path
|
|
$env:SEARCHI_ORT_LIBRARY = (Resolve-Path tools/semantic-proof/.venv/Lib/site-packages/onnxruntime/capi/onnxruntime.dll).Path
|
|
cargo test -p searchi-semantic --test cpu_model -- --ignored --nocapture
|
|
```
|
|
|
|
The smoke test checks actual local inference, vector normalization, English and
|
|
German retrieval, recall at three, and the measured five-of-six top-1 baseline.
|
|
It does not replace the strict production quality gate.
|
|
|
|
The native real-artifact smoke was executed successfully on the Windows VM,
|
|
using that CPU-only DLL. It reproduced five-of-six top-1 and six-of-six recall
|
|
at three. Three default unit tests passed; targeted Clippy passed with warnings
|
|
denied. Linux and macOS native checks have not been executed.
|
|
|
|
The next integration should read bounded `SearchIndex::passage_batch` rows,
|
|
split long passages into overlapping tokenizer windows, preserve source page or
|
|
paragraph locations, and persist each vector with a content fingerprint and
|
|
`MODEL_FINGERPRINT`. Passage IDs are not immutable content identities: changes
|
|
must invalidate stale vectors. Queries should combine lexical and semantic
|
|
retrieval while preserving exact filename priority. Model upgrades build a
|
|
separate vector generation before an atomic swap. Cancellation, persistent
|
|
embedding jobs, ANN retrieval, representative quality corpora, and signed
|
|
cross-platform runtime packaging are still required.
|