searchi/docs/SEMANTIC.md

Local semantic search

Searchi uses Google EmbeddingGemma 2, with the pinned Q8 GGUF text encoder and llama.cpp revision used by DigUp. Production has no MiniLM, simulated-vector, or remote embedding fallback. The rejected ONNX experiment is retained only under the opt-in minilm-proof feature and tools/semantic-proof/HISTORICAL.md.

Model and runtime

• Model: google/embeddinggemma-2, 768-dimensional output.
• GGUF publisher: ggml-org/embeddinggemma-2-GGUF.
• Revision: bfcd298762cc34d0357ece5ebdd31791a3a374d8.
• llama.cpp: b11461, commit 4d756bc72bf00a4aacf410ae15a2d315f3db400d, with DigUp's no-logits memory patch.
• Query prompt: task: search result | query: TEXT.
• Document prompt: title: TITLE_OR_none | text: TEXT.
• Noncausal encoder, mean pooling, finite L2-normalized vectors.
• Query/document token budgets: 512/2048.

The persistent native helper uses bounded NDJSON over stdin/stdout. It has no HTTP server or download code. Rust verifies the text model's length and SHA256 before starting it. Setup provisions assets; search never downloads them.

Production device mode is auto, with available compiled acceleration and CPU fallback. Model/context initialization and returned inference/nonfinite-output failures retry on CPU once. A lost GPU worker can be restarted as CPU for one retry of that request. WorkerInfo and responses report the actual device. Windows VM tests remain CPU only, via SEARCHI_FORCE_CPU=1 and Device::Cpu. GPU execution/failure recovery has not been exercised here.

The Windows worker links a static MSVC CRT and static llama libraries, without an external OpenMP runtime. Production CMake supports cpu, auto, metal, vulkan, and cuda; CPU builds force GPU flags off. Runtime/provider parity and packaging on other operating systems remain independent release checks.

Artifact	Bytes	SHA256
embeddinggemma-2-Q8_0.gguf	309,855,456	2188ac1deca4b77dffefd603c2776a9d76d9d74ec01841392982ebb840b09135
mmproj-embeddinggemma-2-Q8_0.gguf	554,821,024	c4a8a52691ecef40618438928bdf9e68379b854e24166f292592353db0aab64f

Setup downloads both pinned files. The current worker implements text; image/audio projection loading and image/audio/video embeddings remain required work. Downloaded media weights do not establish functioning multimodal search.

Weights use Apache-2.0; llama.cpp and DigUp use MIT; USearch and NumKong use Apache-2.0. Retain notices with distributions. Five DigUp text reference vectors are copied under MIT in crates/searchi-semantic/tests/fixtures with the upstream license and source commit alongside them.

Primary sources:

• Google developer guide
• Model documentation
• Pinned GGUF publisher
• Pinned DigUp reference
• USearch HNSW and filtered retrieval

Shared API and bounds

GemmaEmbedder owns actual tokenization, document/query prompts, and inference. HybridSearch is the shared CLI/desktop service:

HybridSearch::open(db, model_dir, worker_executable, threads)
HybridSearch::embedding_status(db) // No model/runtime load or network
service.warmup()                  // Real query inference + ANN cache
service.index_embeddings()
service.search(query, limit, &SearchOptions)
service.search_semantic(query, limit, &SearchOptions) // Diagnostics
service.worker_info()             // Actual model/runtime/device/threads

Threads are bounded to 1–8; VM tests use one. Document batches contain at most eight windows. Request/response budgets are 2/4 MiB, startup has a 120-second deadline, and each request has a 180-second deadline. Malformed or interrupted IPC and invalid embeddings fail explicitly.

The service makes 64 KiB UTF-8 slices with 512-byte overlap before genuine model token windows. It preserves global byte ranges and page/paragraph/line sources, handles 2 MiB passages, caps each passage at 2048 windows, and bounds titles to 512 UTF-8 bytes. Generation fingerprints include this chunk policy as well as model/runtime/prompts/token budgets/normalization.

Keep a prewarmed query service and a separate background builder. Readiness requires actual inference, rather than merely finding verified model files.

Persistence, ANN, and ranking

Semantic storage has an independent schema version within the shared SQLite database. Chunks store generation, passage/path identity, original-source hash, location, byte range, snippet, and vector. HNSW uses USearch 2.26.4, cosine/F32, 16 connections, construction expansion 128, and search expansion 64. NumKong dispatches supported CPU SIMD kernels with portable fallback. The Cargo config contains the Windows NK_DYNAMIC override required for static NumKong linkage.

Staging vectors and a real HNSW graph are persisted with an ANN blob SHA256. Activation is atomic. Failed rebuilds preserve the active generation; the next rebuild reclaims abandoned staging under a cross-process lock. Search uses a consistent read snapshot.

Metadata filters and source hashes establish eligible keys before ANN top-k. The cache is keyed by generation, corpus revision, and normalized filters. Persistent dirty-path triggers invalidate affected files; unchanged warm queries hash zero passages. Large change batches or expired history rebuild eligibility. Default-filter prewarm includes that initial work.

Fusion uses reciprocal-rank fusion with k = 60. Exact filenames and literal identifiers retain priority. Ordinary partial filenames receive lexical weight 1.15 and can be outranked by keyword/semantic consensus. File results retain source snippets and deterministic path tie breaks.

Current caps are 100,000 chunks and a 512 MiB serialized ANN blob per generation. Full rebuilds still regenerate embeddings and the graph. Incremental reuse/jobs and efficient ANN updates remain production work. One million filename records does not imply one million semantic chunks under these defaults.

Executed Windows CPU evidence

Actual Google CPU vectors match five pinned DigUp text references at cosine 0.99979–0.99994, exceeding the 0.999 gate. These cover English, Bengali, Arabic, a short document, and a document beyond the sliding attention window. Actual tests also cover English/German retrieval, persisted vectors, changed/ deleted-source suppression, verified long windows, and a tail after 64 KiB.

The immutable 20-document/24-query seed corpus produced:

Lane	Top 1	Recall at 3
Lexical	5/24	5/24
Google semantic	24/24	24/24
Hybrid HNSW	24/24	24/24

With one CPU thread, fresh model/context startup took approximately 7 seconds in the background, actual query/ANN prewarm 128 ms, and first foreground natural-language search 99 ms with the correct supplier document first. Warm seed-corpus hybrid p50/p95 was 92/107 ms. Model downloads are excluded.

A separate release-profile benchmark used 100,000 synthetic normalized 768-dimensional vectors and 20 queries against exact nearest neighbors:

Metric	Measurement
Recall at 10 vs exact	98.5%
HNSW query p95	1.37 ms
Warm service retrieval p95	6.40 ms
Warm passages rehashed	0
Initial eligibility work	1.61 seconds
Graph construction/serialization	57.95 seconds
Reported graph memory	554 MB
Dispatched CPU kernel	skylake

This is index-only performance, not neural quality across 100,000 documents. The seed corpus and synthetic vectors do not certify production relevance. Representative corpora, cold/selective-filter latency, concurrent indexing, resource profiles, GPU parity/fallback, multimodal retrieval, and platform releases remain validation gates.

Reproduce

Run from the repository root. Provisioning uses the network explicitly:

python tools/embeddinggemma2/download.py
python tools/embeddinggemma2/build_runtime.py --backend cpu
$env:SEARCHI_FORCE_CPU = '1'
$env:SEARCHI_GEMMA_MODEL_DIR = (Resolve-Path .searchi/models/embeddinggemma2).Path
$env:SEARCHI_GEMMA_WORKER = (Resolve-Path .searchi/runtime/cpu/Release/searchi-gemma-worker.exe).Path
cargo test -p searchi-semantic --test gemma_cpu -- --ignored --nocapture
cargo test -p searchi-service --test google_cpu -- --ignored --nocapture
cargo test -p searchi-service --lib google_cpu_preserves -- --ignored --nocapture
cargo run -p searchi-service --example evaluate -- .searchi/models/embeddinggemma2 .searchi/runtime/cpu/Release/searchi-gemma-worker.exe .searchi/google-relevance-ann.json
$env:SEARCHI_ANN_REPORT = (Join-Path (Get-Location) '.searchi/ann-100k.json')
cargo test --release -p searchi-service ann_100k -- --ignored --nocapture

The ignored ANN test deliberately requires release mode. Default tests require no model/network and cover protocol bounds, invalid vectors, UTF-8 coverage, fusion, lifecycle, checksums, pre-top-k filtering, and dirty-path caching. Linux/macOS and GPU checks are not inferred from this Windows CPU run.