searchi/docs/SEMANTIC.md

Local semantic search

Searchi uses Google EmbeddingGemma 2, with the pinned Q8 GGUF text encoder and llama.cpp revision used by DigUp. Production has no MiniLM, simulated-vector, or remote embedding fallback. The rejected ONNX experiment is retained only under the opt-in minilm-proof feature and tools/semantic-proof/HISTORICAL.md.

Model and runtime

• Model: google/embeddinggemma-2, 768-dimensional output.
• GGUF publisher: ggml-org/embeddinggemma-2-GGUF.
• Revision: bfcd298762cc34d0357ece5ebdd31791a3a374d8.
• llama.cpp: b11461, commit 4d756bc72bf00a4aacf410ae15a2d315f3db400d, with DigUp's no-logits memory patch.
• Query prompt: task: search result | query: TEXT.
• Document prompt: title: TITLE_OR_none | text: TEXT.
• Noncausal encoder, mean pooling, finite L2-normalized vectors.
• Query/document token budgets: 512/2048.

The persistent native helper uses bounded NDJSON over stdin/stdout. It has no HTTP server or download code. Rust verifies the text model's length and SHA256 before starting it. Setup provisions assets; search never downloads them. Each request has one 180-second IPC deadline across pipe writing and response reading. Single-slot queues bound pending work. A stalled write or response kills/reaps the helper and joins both IPC threads; only transport failures qualify for a bounded CPU restart. Malformed/oversized responses remain protocol errors. Cleanup assumes the shipped helper spawns no descendants holding its pipe handles, matching the native implementation.

Production device mode is auto, with available compiled acceleration and CPU fallback. Model/context initialization and returned inference/nonfinite-output failures retry on CPU once. A lost GPU worker can be restarted as CPU for one retry of that request. WorkerInfo and responses report the actual device. Windows VM tests remain CPU only, via SEARCHI_FORCE_CPU=1 and Device::Cpu. GPU execution/failure recovery has not been exercised here.

The normal Windows x64 worker ships shared llama libraries and baseline plus optimized CPU modules. Selection checks CPU features and OS support for AVX register state; it loads packaged backends from the executable directory. Shared libraries use a common MSVC CRT because model file handles cross DLL boundaries. The unchanged release CRT files are packaged alongside the worker. Internal static CPU builds retain a static CRT. Neither profile requires an external OpenMP runtime. Production CMake supports cpu, auto, metal, vulkan, and cuda; CPU builds force GPU flags off. Runtime/provider parity and packaging on other operating systems remain independent release checks.

Linux x86-64 normal builds also use shared CPU dispatch: 14 CPU modules, four core libraries and a dynamically loaded Vulkan module. Core SONAME files are copied beside the worker with origin-relative lookup; the builder verifies ELF dependencies and the stager verifies the explicit hashed inventory. Vulkan's system loader is a dependency of the optional GPU module rather than the worker or CPU/core libraries. Native optimized and baseline-only CPU code-query smoke checks pass. System ABI coverage, reference parity, GPU operation and desktop packaging are separate release gates.

The normal build/staging default is auto, shipping GPU support together with CPU fallback: Metal on macOS, Vulkan on Windows/Linux. Runtime detection chooses the usable device; users do not need hardware-specific builds. Release builders must have the relevant compilation SDK. Missing Vulkan/glslc now fails the normal build rather than silently omitting acceleration. Actual GPU parity, loader/dependency packaging and no-driver CPU fallback still require evidence.

Windows x64 build scripts provision the pinned official Vulkan SDK locally, verify its provenance and stage the app-local loader with its notice. Forced CPU startup disables Vulkan registration before the backend registry initializes; CPU contexts disable device-operation offload. Automatic startup transport failure now restarts once on CPU without weakening model/protocol validation. GPU inference failure also retains the bounded existing CPU retry.

Artifact	Bytes	SHA256
embeddinggemma-2-Q8_0.gguf	309,855,456	2188ac1deca4b77dffefd603c2776a9d76d9d74ec01841392982ebb840b09135
mmproj-embeddinggemma-2-Q8_0.gguf	554,821,024	c4a8a52691ecef40618438928bdf9e68379b854e24166f292592353db0aab64f

Setup downloads both pinned files. The native worker implements genuine text, image pixels, and mono 16 kHz audio embeddings through the verified Google projector in the same 768-dimensional query space. The shared service now persists unified document, image, video-frame and soundtrack/audio/music vectors with source locations and metadata. Scoped desktop media previews and an independent retrieval evaluation are implemented; precise localization, broader quality, native playback QA and releases remain gates.

Weights use Apache-2.0; llama.cpp and DigUp use MIT; USearch and NumKong use Apache-2.0. Retain notices with distributions. Five DigUp text reference vectors are copied under MIT in crates/searchi-semantic/tests/fixtures with the upstream license and source commit alongside them.

Primary sources:

• Google developer guide
• Model documentation
• Pinned GGUF publisher
• Pinned DigUp reference
• USearch HNSW and filtered retrieval

Shared API and bounds

GemmaEmbedder owns actual tokenization, document/query prompts, and inference. HybridSearch is the shared CLI/desktop service:

HybridSearch::open(db, model_dir, worker_executable, threads)
HybridSearch::embedding_status(db) // No model/runtime load or network
service.warmup()                  // Real query inference + ANN cache
service.index_embeddings() // Explicit document-only generation
service.index_embeddings_with_media(&tools, &MediaLimits::default()) // Unified generation
service.search(query, limit, &SearchOptions)
service.search_semantic(query, limit, &SearchOptions) // Diagnostics
service.worker_info()             // Actual model/runtime/device/threads

Threads are bounded to 1–8; VM tests use one. Document batches contain at most eight windows. Request/response budgets are 16/4 MiB, startup has a 120-second deadline, and each request has a 180-second deadline. Malformed or interrupted IPC and invalid embeddings fail explicitly.

The service makes 64 KiB UTF-8 slices with 512-byte overlap before genuine model token windows. It preserves global byte ranges and page/paragraph/line sources, handles 2 MiB passages, caps each passage at 2048 windows, and bounds titles to 512 UTF-8 bytes. Generation fingerprints include this chunk policy as well as model/runtime/prompts/token budgets/normalization.

Keep a prewarmed query service and a separate background builder. Readiness requires actual inference, rather than merely finding verified model files.

Persistence, ANN, and ranking

Semantic storage has an independent schema version within the shared SQLite database. Chunks store generation, passage/path identity, original-source hash, location, byte range, snippet, and vector. HNSW uses USearch 2.26.4, cosine/F32, 16 connections, construction expansion 128, and search expansion 64. NumKong dispatches supported CPU SIMD kernels with portable fallback. The Cargo config contains the Windows NK_DYNAMIC override required for static NumKong linkage.

Staging vectors and a real HNSW graph are persisted with an ANN blob SHA256. Activation is atomic. Failed rebuilds preserve the active generation and the same search session's reader policy, including failures acquiring the build lock, starting a generation, publishing it or discarding failed staging. The original failure remains visible if cleanup also fails. The next rebuild reclaims abandoned staging under a cross-process lock. Search uses a consistent read snapshot.

Metadata filters and source hashes establish eligible keys before ANN top-k. The cache is keyed by generation, corpus revision, and normalized filters. Persistent dirty-path triggers invalidate affected files; unchanged warm queries hash zero passages. Large change batches or expired history rebuild eligibility. Default-filter prewarm includes that initial work.

Fusion uses reciprocal-rank fusion with k = 60. Exact filenames and literal identifiers retain priority. Ordinary partial filenames receive lexical weight 1.15 and can be outranked by keyword/semantic consensus. File results retain source snippets and deterministic path tie breaks.

Current caps are 100,000 chunks and a 512 MiB serialized ANN blob per generation. Full rebuilds still regenerate embeddings and the graph. Incremental reuse/jobs and efficient ANN updates remain production work. One million filename records does not imply one million semantic chunks under these defaults.

Executed Windows CPU evidence

Actual Google CPU vectors match five pinned DigUp text references at cosine 0.99979–0.99994, exceeding the 0.999 gate. These cover English, Bengali, Arabic, a short document, and a document beyond the sliding attention window. Actual tests also cover English/German retrieval, persisted vectors, changed/ deleted-source suppression, verified long windows, and a tail after 64 KiB.

The immutable 20-document/24-query seed corpus produced:

Lane	Top 1	Recall at 3
Lexical	5/24	5/24
Google semantic	24/24	24/24
Hybrid HNSW	24/24	24/24

With one CPU thread, fresh model/context startup took approximately 7 seconds in the background, actual query/ANN prewarm 128 ms, and first foreground natural-language search 99 ms with the correct supplier document first. Warm seed-corpus hybrid p50/p95 was 92/107 ms. Model downloads are excluded.

A separate release-profile benchmark used 100,000 synthetic normalized 768-dimensional vectors and 20 queries against exact nearest neighbors:

Metric	Measurement
Recall at 10 vs exact	98.5%
HNSW query p95	1.37 ms
Warm service retrieval p95	6.40 ms
Warm passages rehashed	0
Initial eligibility work	1.61 seconds
Graph construction/serialization	57.95 seconds
Reported graph memory	554 MB
Dispatched CPU kernel	skylake

This is index-only performance, not neural quality across 100,000 documents. The seed corpus and synthetic vectors do not certify production relevance. Representative corpora, cold/selective-filter latency, concurrent indexing, resource profiles, GPU parity/fallback, multimodal retrieval, and platform releases remain validation gates.

Reproduce

Run from the repository root. Provisioning uses the network explicitly:

python tools/embeddinggemma2/download.py
python tools/embeddinggemma2/build_runtime.py --backend cpu
$env:SEARCHI_FORCE_CPU = '1'
$env:SEARCHI_GEMMA_MODEL_DIR = (Resolve-Path .searchi/models/embeddinggemma2).Path
$env:SEARCHI_GEMMA_WORKER = (Resolve-Path .searchi/runtime/cpu/Release/searchi-gemma-worker.exe).Path
cargo test -p searchi-semantic --test gemma_cpu -- --ignored --nocapture
cargo test -p searchi-service --test google_cpu -- --ignored --nocapture
cargo test -p searchi-service --lib google_cpu_preserves -- --ignored --nocapture
cargo run -p searchi-service --example evaluate -- .searchi/models/embeddinggemma2 .searchi/runtime/cpu/Release/searchi-gemma-worker.exe .searchi/google-relevance-ann.json
$env:SEARCHI_ANN_REPORT = (Join-Path (Get-Location) '.searchi/ann-100k.json')
cargo test --release -p searchi-service ann_100k -- --ignored --nocapture

The ignored ANN test deliberately requires release mode. Default tests require no model/network and cover protocol bounds, invalid vectors, UTF-8 coverage, fusion, lifecycle, checksums, pre-top-k filtering, and dirty-path caching. Linux/macOS and GPU checks are not inferred from this Windows CPU run.

Genuine image and audio adapter

GemmaEmbedder::embed_images(&[&Path]) uses a provisional 280-token image budget. embed_images_with_budget(paths, budget) supports 70 through 1120 Google visual tokens, including full-resolution model input. Use embed_images_with_metadata for encoded dimensions and the chosen budget. The aspect-preserving dimensions are multiples of 48; Rust performs bounded PNG/JPEG/WebP/BMP decoding, applies EXIF orientation, and bicubic Catmull-Rom resizing before sending RGB pixels. No captions or filenames produce vectors. Bounds are eight paths per call, one image per inference operation, 64 MiB input files, 64 Mi-pixel JPEG sources (scaled decoding above 16 Mi pixels), 16 Mi-pixel non-JPEG sources, and a 256 MiB decoding-buffer/output ceiling. Decoder dimensions and output size are checked before pixel allocation. Tiny/extreme images still receive a valid bounded grid. The same API embeds decoded video frame files.

GemmaEmbedder::embed_audio_pcm(samples) accepts normalized finite mono 16 kHz PCM with 1–480,000 samples, at most 30 seconds, on a little-endian host. It encodes actual sound through Google MTMD and the shared mean-pooled text model. Callers must segment the entire recording/movie soundtrack and retain actual window timestamps; this API never silently truncates audio. It provides no transcript. Speech, music, and environmental sounds require independent quality evaluation even though all use this genuine encoder.

The projector is size/SHA256 verified on first media use, then loaded lazily in the retained native worker. Model initialization and returned inference failures share the CPU fallback policy. Native video/FFmpeg spawning is disabled; the media adapter owns extraction. Media generations must include GEMMA_MEDIA_FINGERPRINT and image budget/window policy in their identity. Text fingerprints remain unchanged. GPU media inference is untested here.

Actual Windows CPU parity with copied MIT DigUp fixtures passed for screenshot 280, photo 140, frame 70, and speech: cosine respectively 0.99994194, 0.99974583, 0.99921928, and 0.99983318, all exceeding the 0.999 gate. The test completed in 58.79 seconds including fresh model/projector startup. This proves runtime parity on those inputs, not broad image/audio retrieval quality. Run with the model/worker environment above:

cargo test -p searchi-semantic --test gemma_media_cpu -- --ignored --nocapture

The large camera JPEG policy, established decoder tradeoffs, EXIF/resource test, and memory-bound caveats are documented in the semantic crate.

Unified media indexing and setup

The desktop and default CLI embed command use index_embeddings_with_media. Document-only indexing is an explicit --text-only choice. searchi-media prepares actual image files, movie frames and mono 16 kHz audio windows with verified caller-supplied FFmpeg/FFprobe executables. There is no transcription, fabricated caption or model-derived filename substitution. Existing stream/file artist/title metadata enters guarded keyword indexing.

Default media policy uses image 280/frame 70 visual tokens, 30-second primary audio windows with two-second overlap, up to 240 distributed frames plus 120 scene candidates, up to 600 audio windows, a four-hour/16 GiB file ceiling and 45-second subprocess deadlines. Coverage includes the audio tail or reports an explicit resource error; video sampling spans the movie without guaranteeing every transient scene. Media files have stable content identity, persisted vectors and sources (image, timestamp_ms); source changes/deletions suppress stale results. Generations include preprocessing/window/sampling fingerprints. Per-file decoding failures are reported and persisted rather than silently counted as successful media indexing. Other filename/text matches stay useful.

Developer preparation from the repository root:

python tools/embeddinggemma2/build_runtime.py --backend cpu --jobs 2
python tools/embeddinggemma2/stage_runtime.py --backend cpu
python tools/media-runtime/provision.py --stage
$env:SEARCHI_FORCE_CPU = '1'
$env:WEBVIEW2_ADDITIONAL_BROWSER_ARGUMENTS = '--disable-gpu --disable-accelerated-video-decode'

The decoder script verifies a pinned FFmpeg 9.0.2 LGPL artifact and stages tools, notices and the manifest without global installation. Only Windows x86-64 has been executed; Linux/ARM64 entries require native checks, and macOS decoder packaging is still missing. End-user packages must include the worker/decoders; Google weights are provisioned automatically and verified locally. Search does not perform setup downloads. Headless CLI helper discovery and explicit paths are documented in README and BUILDING.

Independent media evaluation preserves the frozen corpus, actual production pipeline, direct-harness failure, music confusions, speech point-localization failures and unsuppressed no-answer results. Strong file recall is insufficient for accurate spoken-topic navigation. A ten-second audio experiment reduces some distances but does not close that gate; it is not a new default. Quality policies must be validated on held-out data rather than inventing thresholds from the evaluation corpus.