karl
caf6a2f82e
Core CI / core (ubuntu-latest) (push) Failing after 3m27s
Core CI / core (macos-latest) (push) Canceled after 0s
Core CI / core (windows-latest) (push) Canceled after 0s
Core CI / desktop (ubuntu-latest) (push) Successful in 5m45s
Core CI / desktop (macos-latest) (push) Canceled after 0s
Core CI / desktop (windows-latest) (push) Canceled after 0s
180 lines
9.1 KiB
Markdown
180 lines
9.1 KiB
Markdown
# Local semantic search
|
||
|
||
Searchi uses **Google EmbeddingGemma 2**, with the pinned Q8 GGUF text encoder
|
||
and llama.cpp revision used by DigUp. Production has no MiniLM, simulated-vector,
|
||
or remote embedding fallback. The rejected ONNX experiment is retained only
|
||
under the opt-in minilm-proof feature and tools/semantic-proof/HISTORICAL.md.
|
||
|
||
## Model and runtime
|
||
|
||
- Model: google/embeddinggemma-2, **768-dimensional** output.
|
||
- GGUF publisher: ggml-org/embeddinggemma-2-GGUF.
|
||
- Revision: bfcd298762cc34d0357ece5ebdd31791a3a374d8.
|
||
- llama.cpp: b11461, commit 4d756bc72bf00a4aacf410ae15a2d315f3db400d,
|
||
with DigUp's no-logits memory patch.
|
||
- Query prompt: task: search result | query: TEXT.
|
||
- Document prompt: title: TITLE_OR_none | text: TEXT.
|
||
- Noncausal encoder, mean pooling, finite L2-normalized vectors.
|
||
- Query/document token budgets: 512/2048.
|
||
|
||
The persistent native helper uses bounded NDJSON over stdin/stdout. It has no
|
||
HTTP server or download code. Rust verifies the text model's length and SHA256
|
||
before starting it. Setup provisions assets; search never downloads them.
|
||
|
||
Production device mode is auto, with available compiled acceleration and CPU
|
||
fallback. Model/context initialization and returned inference/nonfinite-output
|
||
failures retry on CPU once. A lost GPU worker can be restarted as CPU for one
|
||
retry of that request. WorkerInfo and responses report the actual device.
|
||
**Windows VM tests remain CPU only**, via SEARCHI_FORCE_CPU=1 and Device::Cpu.
|
||
GPU execution/failure recovery has not been exercised here.
|
||
|
||
The Windows worker links a static MSVC CRT and static llama libraries, without
|
||
an external OpenMP runtime. Production CMake supports cpu, auto, metal, vulkan,
|
||
and cuda; CPU builds force GPU flags off. Runtime/provider parity and packaging
|
||
on other operating systems remain independent release checks.
|
||
|
||
| Artifact | Bytes | SHA256 |
|
||
| --- | ---: | --- |
|
||
| embeddinggemma-2-Q8_0.gguf | 309,855,456 | 2188ac1deca4b77dffefd603c2776a9d76d9d74ec01841392982ebb840b09135 |
|
||
| mmproj-embeddinggemma-2-Q8_0.gguf | 554,821,024 | c4a8a52691ecef40618438928bdf9e68379b854e24166f292592353db0aab64f |
|
||
|
||
Setup downloads both pinned files. **The current worker implements text**;
|
||
image/audio projection loading and image/audio/video embeddings remain required
|
||
work. Downloaded media weights do not establish functioning multimodal search.
|
||
|
||
Weights use Apache-2.0; llama.cpp and DigUp use MIT; USearch and NumKong use
|
||
Apache-2.0. Retain notices with distributions. Five DigUp text reference vectors
|
||
are copied under MIT in crates/searchi-semantic/tests/fixtures with the upstream
|
||
license and source commit alongside them.
|
||
|
||
Primary sources:
|
||
|
||
- [Google developer guide](https://developers.googleblog.com/embeddinggemma-2-the-developer-guide/)
|
||
- [Model documentation](https://huggingface.co/docs/transformers/model_doc/embedding_gemma2)
|
||
- [Pinned GGUF publisher](https://huggingface.co/ggml-org/embeddinggemma-2-GGUF/tree/bfcd298762cc34d0357ece5ebdd31791a3a374d8)
|
||
- [Pinned DigUp reference](https://github.com/ARahim3/DigUp/tree/6faa29e4c315a3a591de1b41bd78751bdf39588d)
|
||
- [USearch HNSW and filtered retrieval](https://github.com/unum-cloud/USearch)
|
||
|
||
## Shared API and bounds
|
||
|
||
GemmaEmbedder owns actual tokenization, document/query prompts, and inference.
|
||
HybridSearch is the shared CLI/desktop service:
|
||
|
||
~~~rust
|
||
HybridSearch::open(db, model_dir, worker_executable, threads)
|
||
HybridSearch::embedding_status(db) // No model/runtime load or network
|
||
service.warmup() // Real query inference + ANN cache
|
||
service.index_embeddings()
|
||
service.search(query, limit, &SearchOptions)
|
||
service.search_semantic(query, limit, &SearchOptions) // Diagnostics
|
||
service.worker_info() // Actual model/runtime/device/threads
|
||
~~~
|
||
|
||
Threads are bounded to 1–8; VM tests use one. Document batches contain at most
|
||
eight windows. Request/response budgets are 2/4 MiB, startup has a 120-second
|
||
deadline, and each request has a 180-second deadline. Malformed or interrupted
|
||
IPC and invalid embeddings fail explicitly.
|
||
|
||
The service makes 64 KiB UTF-8 slices with 512-byte overlap before genuine model
|
||
token windows. It preserves global byte ranges and page/paragraph/line sources,
|
||
handles 2 MiB passages, caps each passage at 2048 windows, and bounds titles to
|
||
512 UTF-8 bytes. Generation fingerprints include this chunk policy as well as
|
||
model/runtime/prompts/token budgets/normalization.
|
||
|
||
Keep a prewarmed query service and a separate background builder. Readiness
|
||
requires actual inference, rather than merely finding verified model files.
|
||
|
||
## Persistence, ANN, and ranking
|
||
|
||
Semantic storage has an independent schema version within the shared SQLite
|
||
database. Chunks store generation, passage/path identity, original-source hash,
|
||
location, byte range, snippet, and vector. HNSW uses USearch 2.26.4, cosine/F32,
|
||
16 connections, construction expansion 128, and search expansion 64. NumKong
|
||
dispatches supported CPU SIMD kernels with portable fallback. The Cargo config
|
||
contains the Windows NK_DYNAMIC override required for static NumKong linkage.
|
||
|
||
Staging vectors and a real HNSW graph are persisted with an ANN blob SHA256.
|
||
Activation is atomic. Failed rebuilds preserve the active generation; the next
|
||
rebuild reclaims abandoned staging under a cross-process lock. Search uses a
|
||
consistent read snapshot.
|
||
|
||
Metadata filters and source hashes establish eligible keys before ANN top-k.
|
||
The cache is keyed by generation, corpus revision, and normalized filters.
|
||
Persistent dirty-path triggers invalidate affected files; unchanged warm
|
||
queries hash zero passages. Large change batches or expired history rebuild
|
||
eligibility. Default-filter prewarm includes that initial work.
|
||
|
||
Fusion uses reciprocal-rank fusion with k = 60. Exact filenames and literal
|
||
identifiers retain priority. Ordinary partial filenames receive lexical
|
||
weight 1.15 and can be outranked by keyword/semantic consensus. File results
|
||
retain source snippets and deterministic path tie breaks.
|
||
|
||
Current caps are 100,000 chunks and a 512 MiB serialized ANN blob per generation.
|
||
Full rebuilds still regenerate embeddings and the graph. Incremental reuse/jobs
|
||
and efficient ANN updates remain production work. One million filename records
|
||
does not imply one million semantic chunks under these defaults.
|
||
|
||
## Executed Windows CPU evidence
|
||
|
||
Actual Google CPU vectors match five pinned DigUp text references at cosine
|
||
**0.99979–0.99994**, exceeding the 0.999 gate. These cover English, Bengali,
|
||
Arabic, a short document, and a document beyond the sliding attention window.
|
||
Actual tests also cover English/German retrieval, persisted vectors, changed/
|
||
deleted-source suppression, verified long windows, and a tail **after 64 KiB**.
|
||
|
||
The immutable 20-document/24-query seed corpus produced:
|
||
|
||
| Lane | Top 1 | Recall at 3 |
|
||
| --- | ---: | ---: |
|
||
| Lexical | 5/24 | 5/24 |
|
||
| Google semantic | 24/24 | 24/24 |
|
||
| Hybrid HNSW | 24/24 | 24/24 |
|
||
|
||
With one CPU thread, fresh model/context startup took approximately **7 seconds
|
||
in the background**, actual query/ANN prewarm **128 ms**, and first foreground
|
||
natural-language search **99 ms** with the correct supplier document first.
|
||
Warm seed-corpus hybrid p50/p95 was **92/107 ms**. Model downloads are excluded.
|
||
|
||
A separate release-profile benchmark used 100,000 synthetic normalized 768-dimensional
|
||
vectors and 20 queries against exact nearest neighbors:
|
||
|
||
| Metric | Measurement |
|
||
| --- | ---: |
|
||
| Recall at 10 vs exact | 98.5% |
|
||
| HNSW query p95 | 1.37 ms |
|
||
| Warm service retrieval p95 | 6.40 ms |
|
||
| Warm passages rehashed | 0 |
|
||
| Initial eligibility work | 1.61 seconds |
|
||
| Graph construction/serialization | 57.95 seconds |
|
||
| Reported graph memory | 554 MB |
|
||
| Dispatched CPU kernel | skylake |
|
||
|
||
This is **index-only performance**, not neural quality across 100,000 documents.
|
||
The seed corpus and synthetic vectors do not certify production relevance.
|
||
Representative corpora, cold/selective-filter latency, concurrent indexing,
|
||
resource profiles, GPU parity/fallback, multimodal retrieval, and platform
|
||
releases remain validation gates.
|
||
|
||
## Reproduce
|
||
|
||
Run from the repository root. Provisioning uses the network explicitly:
|
||
|
||
~~~powershell
|
||
python tools/embeddinggemma2/download.py
|
||
python tools/embeddinggemma2/build_runtime.py --backend cpu
|
||
$env:SEARCHI_FORCE_CPU = '1'
|
||
$env:SEARCHI_GEMMA_MODEL_DIR = (Resolve-Path .searchi/models/embeddinggemma2).Path
|
||
$env:SEARCHI_GEMMA_WORKER = (Resolve-Path .searchi/runtime/cpu/Release/searchi-gemma-worker.exe).Path
|
||
cargo test -p searchi-semantic --test gemma_cpu -- --ignored --nocapture
|
||
cargo test -p searchi-service --test google_cpu -- --ignored --nocapture
|
||
cargo test -p searchi-service --lib google_cpu_preserves -- --ignored --nocapture
|
||
cargo run -p searchi-service --example evaluate -- .searchi/models/embeddinggemma2 .searchi/runtime/cpu/Release/searchi-gemma-worker.exe .searchi/google-relevance-ann.json
|
||
$env:SEARCHI_ANN_REPORT = (Join-Path (Get-Location) '.searchi/ann-100k.json')
|
||
cargo test --release -p searchi-service ann_100k -- --ignored --nocapture
|
||
~~~
|
||
|
||
The ignored ANN test deliberately requires release mode. Default tests require
|
||
no model/network and cover protocol bounds, invalid vectors, UTF-8 coverage,
|
||
fusion, lifecycle, checksums, pre-top-k filtering, and dirty-path caching.
|
||
Linux/macOS and GPU checks are not inferred from this Windows CPU run.
|