searchi/docs/SEMANTIC.md

CPU semantic inference proof

This milestone runs genuine neural embeddings locally. It does not yet connect semantic vectors to the SQLite search index or desktop interface. Production retrieval quality, large-corpus performance, and Linux/macOS runtime packaging remain unverified.

Model and runtime decision

The first working baseline is the quantized ONNX export of sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2, published by Xenova. The upstream model card describes 384-dimensional text embeddings, attention-mask mean pooling, a 128-token maximum, and Apache-2.0 licensing. It covers multiple languages. This is a text-only baseline; it does not claim EmbeddingGemma 2's multimodal features or compatibility.

Sources reviewed on 2026-10-12:

• Original model card and pooling architecture
• ONNX export publisher
• ONNX Runtime Python session API
• Rust ort dynamic-library initialization
• Rust ort session configuration

Inference explicitly selects CPUExecutionProvider. Python uses the CPU onnxruntime wheel, never onnxruntime-gpu. The native Rust crate disables ort default features: there are no binary downloads, GPU provider features, or model HTTP client features. It loads an explicitly supplied local runtime library and disables runtime telemetry. Tokenization is sequential; ONNX sessions use sequential graph execution with bounded intra-op threads and one inter-op thread.

Reproduce on Windows

Run these commands from the repository root. Provisioning commands intentionally use the network; the inference command does not provision anything.

python -m venv tools/semantic-proof/.venv
& tools/semantic-proof/.venv/Scripts/python.exe -m pip install --only-binary=:all: -r tools/semantic-proof/requirements.txt
& tools/semantic-proof/.venv/Scripts/python.exe tools/semantic-proof/download.py
& tools/semantic-proof/.venv/Scripts/python.exe tools/semantic-proof/proof.py --threads 1 --output tools/semantic-proof/windows-cpu-result.json

On Linux/macOS, use .venv/bin/python instead of .venv/Scripts/python.exe. These instructions have only been executed on Windows. Wheel availability and native library distribution must be checked on other platforms.

download.py downloads exactly two revision-pinned artifacts, streams them to a temporary file, rejects oversized downloads, checks byte length and SHA256, then atomically replaces the local asset. Assets occupy approximately 136 MB in decimal units and are ignored by Git. Inference validates both files again and fails clearly when either is missing or damaged. It never contacts the model hub. The Python proof sets offline environment flags and rejects Python socket connect, name-resolution, and bind audit events throughout import and inference. This guard checks Python socket activity; it is not an operating-system firewall against arbitrary native libraries.

The proof intentionally exits nonzero if any fixture query misses top-1. Its JSON report is written before the quality gate, preserving failures for review.

Pinned artifacts and licensing

Revision: 2c4055b12046f11709e9df2c122e59ffbdc2f900. The manifest in tools/semantic-proof/model-manifest.json records the same model contract and hashes used by the native adapter.

Artifact	Bytes	SHA256
onnx/model_quantized.onnx	118,308,126	66fc00f5f29afcaff34092e1bdd20008ca3918265a82fb9695a551e510cc4ebc
tokenizer.json	17,082,913	b60b6b43406a48bf3638526314f3d232d97058bc93472ff2de930d43686fa441

The base model card specifies Apache-2.0. The export card identifies that base model but does not display its own license declaration. Before redistributing the ONNX export in installers, confirm export licensing and preserve the base license, attribution, and applicable notices. Current code explicitly downloads user-local assets and does not ship model weights or runtime DLLs in Git. ONNX Runtime uses MIT licensing; ort uses MIT or Apache-2.0 and tokenizers uses Apache-2.0. Release packaging must retain third-party notices and validate platform-specific runtime hashes independently; a model hash does not attest to the runtime library.

The tested Windows ONNX Runtime 1.31.0 DLL is 19,288,888 bytes with SHA256 90497e3d406c09f4d59d364c696a2f93dc02f6ef13741b3b89eec95bdb4a3806. The adapter currently trusts the explicitly supplied runtime path; it validates model hashes, not runtime-library hashes. A release distribution must add a platform runtime manifest before shipping. requirements-lock-windows.txt captures the exact Python environment used for this proof.

Measured Windows VM result

tools/semantic-proof/windows-cpu-result.json contains the full measurement and rankings. The run used Windows 11 AMD64, Python 3.14.8, ONNX Runtime 1.31.0, four visible logical CPUs, and one inference thread with only the CPU provider.

Metric	Measurement
Fixture documents / queries	8 / 6
Timed warm query runs	60
Artifact verification plus runtime/model load	approximately 3.4 seconds
Corpus batch embedding	approximately 145 ms
Warm query p50 / p95	approximately 3.1 / 11.9 ms
Top-1 accuracy	5/6 (83.3%)
Recall at 3	6/6 (100%)

The corpus includes English and German documents, same-language semantic queries, and cross-language queries. A question about a postponed product release ranks the supplier contract above the expected meeting notes. That failure remains in the fixture and blocks the strict top-1 quality gate. This small demonstration supports the feasibility of CPU inference; it does not establish large-corpus latency, robust multilingual quality, or production readiness. Quantization and batch shape can affect similarity scores.

Native integration contract

searchi-semantic exposes:

CpuEmbedder::load(model_dir, runtime_library, threads) -> Result<CpuEmbedder>
CpuEmbedder::encode(&mut self, texts: &[&str]) -> Result<Vec<Vec<f32>>>
DIMENSIONS // 384
MODEL_FINGERPRINT // revision + quantization + pooling + normalization + token cap

Inputs are bounded to eight texts per batch, 64 KiB UTF-8 per text, and 128 tokens. Outputs are finite, normalized f32 vectors. The runtime library is process-global; adapter initialization rejects attempts to switch libraries or silently accept an environment initialized outside the adapter. Supply a known, trusted runtime before calling any other ort APIs in the process.

Default unit tests need no model or runtime. The real-artifact test is opt-in:

$env:SEARCHI_MODEL_DIR = (Resolve-Path tools/semantic-proof/model).Path
$env:SEARCHI_ORT_LIBRARY = (Resolve-Path tools/semantic-proof/.venv/Lib/site-packages/onnxruntime/capi/onnxruntime.dll).Path
cargo test -p searchi-semantic --test cpu_model -- --ignored --nocapture

The smoke test checks actual local inference, vector normalization, English and German retrieval, recall at three, and the measured five-of-six top-1 baseline. It does not replace the strict production quality gate.

The native real-artifact smoke was executed successfully on the Windows VM, using that CPU-only DLL. It reproduced five-of-six top-1 and six-of-six recall at three. Three default unit tests passed; targeted Clippy passed with warnings denied. Linux and macOS native checks have not been executed.

The next integration should read bounded SearchIndex::passage_batch rows, split long passages into overlapping tokenizer windows, preserve source page or paragraph locations, and persist each vector with a content fingerprint and MODEL_FINGERPRINT. Passage IDs are not immutable content identities: changes must invalidate stale vectors. Queries should combine lexical and semantic retrieval while preserving exact filename priority. Model upgrades build a separate vector generation before an atomic swap. Cancellation, persistent embedding jobs, ANN retrieval, representative quality corpora, and signed cross-platform runtime packaging are still required.