searchi/docs/SEMANTIC.md

# Local semantic search

Searchi uses **Google EmbeddingGemma 2**, with the pinned Q8 GGUF text encoder
and llama.cpp revision used by DigUp. Production has no MiniLM, simulated-vector,
or remote embedding fallback. The rejected ONNX experiment is retained only
under the opt-in minilm-proof feature and tools/semantic-proof/HISTORICAL.md.

## Model and runtime

- Model: google/embeddinggemma-2, **768-dimensional** output.
- GGUF publisher: ggml-org/embeddinggemma-2-GGUF.
- Revision: bfcd298762cc34d0357ece5ebdd31791a3a374d8.
- llama.cpp: b11461, commit 4d756bc72bf00a4aacf410ae15a2d315f3db400d,
  with DigUp's no-logits memory patch.
- Query prompt: task: search result | query: TEXT.
- Document prompt: title: TITLE_OR_none | text: TEXT.
- Noncausal encoder, mean pooling, finite L2-normalized vectors.
- Query/document token budgets: 512/2048.

The persistent native helper uses bounded NDJSON over stdin/stdout. It has no
HTTP server or download code. Rust verifies the text model's length and SHA256
before starting it. Setup provisions assets; search never downloads them.

Production device mode is auto, with available compiled acceleration and CPU
fallback. Model/context initialization and returned inference/nonfinite-output
failures retry on CPU once. A lost GPU worker can be restarted as CPU for one
retry of that request. WorkerInfo and responses report the actual device.
**Windows VM tests remain CPU only**, via SEARCHI_FORCE_CPU=1 and Device::Cpu.
GPU execution/failure recovery has not been exercised here.

The Windows worker links a static MSVC CRT and static llama libraries, without
an external OpenMP runtime. Production CMake supports cpu, auto, metal, vulkan,
and cuda; CPU builds force GPU flags off. Runtime/provider parity and packaging
on other operating systems remain independent release checks.

| Artifact | Bytes | SHA256 |
| --- | ---: | --- |
| embeddinggemma-2-Q8_0.gguf | 309,855,456 | 2188ac1deca4b77dffefd603c2776a9d76d9d74ec01841392982ebb840b09135 |
| mmproj-embeddinggemma-2-Q8_0.gguf | 554,821,024 | c4a8a52691ecef40618438928bdf9e68379b854e24166f292592353db0aab64f |

Setup downloads both pinned files. **The current worker implements text**;
image/audio projection loading and image/audio/video embeddings remain required
work. Downloaded media weights do not establish functioning multimodal search.

Weights use Apache-2.0; llama.cpp and DigUp use MIT; USearch and NumKong use
Apache-2.0. Retain notices with distributions. Five DigUp text reference vectors
are copied under MIT in crates/searchi-semantic/tests/fixtures with the upstream
license and source commit alongside them.

Primary sources:

- [Google developer guide](https://developers.googleblog.com/embeddinggemma-2-the-developer-guide/)
- [Model documentation](https://huggingface.co/docs/transformers/model_doc/embedding_gemma2)
- [Pinned GGUF publisher](https://huggingface.co/ggml-org/embeddinggemma-2-GGUF/tree/bfcd298762cc34d0357ece5ebdd31791a3a374d8)
- [Pinned DigUp reference](https://github.com/ARahim3/DigUp/tree/6faa29e4c315a3a591de1b41bd78751bdf39588d)
- [USearch HNSW and filtered retrieval](https://github.com/unum-cloud/USearch)

## Shared API and bounds

GemmaEmbedder owns actual tokenization, document/query prompts, and inference.
HybridSearch is the shared CLI/desktop service:

~~~rust
HybridSearch::open(db, model_dir, worker_executable, threads)
HybridSearch::embedding_status(db) // No model/runtime load or network
service.warmup()                  // Real query inference + ANN cache
service.index_embeddings()
service.search(query, limit, &SearchOptions)
service.search_semantic(query, limit, &SearchOptions) // Diagnostics
service.worker_info()             // Actual model/runtime/device/threads
~~~

Threads are bounded to 1–8; VM tests use one. Document batches contain at most
eight windows. Request/response budgets are 2/4 MiB, startup has a 120-second
deadline, and each request has a 180-second deadline. Malformed or interrupted
IPC and invalid embeddings fail explicitly.

The service makes 64 KiB UTF-8 slices with 512-byte overlap before genuine model
token windows. It preserves global byte ranges and page/paragraph/line sources,
handles 2 MiB passages, caps each passage at 2048 windows, and bounds titles to
512 UTF-8 bytes. Generation fingerprints include this chunk policy as well as
model/runtime/prompts/token budgets/normalization.

Keep a prewarmed query service and a separate background builder. Readiness
requires actual inference, rather than merely finding verified model files.

## Persistence, ANN, and ranking

Semantic storage has an independent schema version within the shared SQLite
database. Chunks store generation, passage/path identity, original-source hash,
location, byte range, snippet, and vector. HNSW uses USearch 2.26.4, cosine/F32,
16 connections, construction expansion 128, and search expansion 64. NumKong
dispatches supported CPU SIMD kernels with portable fallback. The Cargo config
contains the Windows NK_DYNAMIC override required for static NumKong linkage.

Staging vectors and a real HNSW graph are persisted with an ANN blob SHA256.
Activation is atomic. Failed rebuilds preserve the active generation; the next
rebuild reclaims abandoned staging under a cross-process lock. Search uses a
consistent read snapshot.

Metadata filters and source hashes establish eligible keys before ANN top-k.
The cache is keyed by generation, corpus revision, and normalized filters.
Persistent dirty-path triggers invalidate affected files; unchanged warm
queries hash zero passages. Large change batches or expired history rebuild
eligibility. Default-filter prewarm includes that initial work.

Fusion uses reciprocal-rank fusion with k = 60. Exact filenames and literal
identifiers retain priority. Ordinary partial filenames receive lexical
weight 1.15 and can be outranked by keyword/semantic consensus. File results
retain source snippets and deterministic path tie breaks.

Current caps are 100,000 chunks and a 512 MiB serialized ANN blob per generation.
Full rebuilds still regenerate embeddings and the graph. Incremental reuse/jobs
and efficient ANN updates remain production work. One million filename records
does not imply one million semantic chunks under these defaults.

## Executed Windows CPU evidence

Actual Google CPU vectors match five pinned DigUp text references at cosine
**0.99979–0.99994**, exceeding the 0.999 gate. These cover English, Bengali,
Arabic, a short document, and a document beyond the sliding attention window.
Actual tests also cover English/German retrieval, persisted vectors, changed/
deleted-source suppression, verified long windows, and a tail **after 64 KiB**.

The immutable 20-document/24-query seed corpus produced:

| Lane | Top 1 | Recall at 3 |
| --- | ---: | ---: |
| Lexical | 5/24 | 5/24 |
| Google semantic | 24/24 | 24/24 |
| Hybrid HNSW | 24/24 | 24/24 |

With one CPU thread, fresh model/context startup took approximately **7 seconds
in the background**, actual query/ANN prewarm **128 ms**, and first foreground
natural-language search **99 ms** with the correct supplier document first.
Warm seed-corpus hybrid p50/p95 was **92/107 ms**. Model downloads are excluded.

A separate release-profile benchmark used 100,000 synthetic normalized 768-dimensional
vectors and 20 queries against exact nearest neighbors:

| Metric | Measurement |
| --- | ---: |
| Recall at 10 vs exact | 98.5% |
| HNSW query p95 | 1.37 ms |
| Warm service retrieval p95 | 6.40 ms |
| Warm passages rehashed | 0 |
| Initial eligibility work | 1.61 seconds |
| Graph construction/serialization | 57.95 seconds |
| Reported graph memory | 554 MB |
| Dispatched CPU kernel | skylake |

This is **index-only performance**, not neural quality across 100,000 documents.
The seed corpus and synthetic vectors do not certify production relevance.
Representative corpora, cold/selective-filter latency, concurrent indexing,
resource profiles, GPU parity/fallback, multimodal retrieval, and platform
releases remain validation gates.

## Reproduce

Run from the repository root. Provisioning uses the network explicitly:

~~~powershell
python tools/embeddinggemma2/download.py
python tools/embeddinggemma2/build_runtime.py --backend cpu
$env:SEARCHI_FORCE_CPU = '1'
$env:SEARCHI_GEMMA_MODEL_DIR = (Resolve-Path .searchi/models/embeddinggemma2).Path
$env:SEARCHI_GEMMA_WORKER = (Resolve-Path .searchi/runtime/cpu/Release/searchi-gemma-worker.exe).Path
cargo test -p searchi-semantic --test gemma_cpu -- --ignored --nocapture
cargo test -p searchi-service --test google_cpu -- --ignored --nocapture
cargo test -p searchi-service --lib google_cpu_preserves -- --ignored --nocapture
cargo run -p searchi-service --example evaluate -- .searchi/models/embeddinggemma2 .searchi/runtime/cpu/Release/searchi-gemma-worker.exe .searchi/google-relevance-ann.json
$env:SEARCHI_ANN_REPORT = (Join-Path (Get-Location) '.searchi/ann-100k.json')
cargo test --release -p searchi-service ann_100k -- --ignored --nocapture
~~~

The ignored ANN test deliberately requires release mode. Default tests require
no model/network and cover protocol bounds, invalid vectors, UTF-8 coverage,
fusion, lifecycle, checksums, pre-top-k filtering, and dirty-path caching.
Linux/macOS and GPU checks are not inferred from this Windows CPU run.